Mercurial > hg
annotate mercurial/worker.py @ 47072:4c041c71ec01
revlog: introduce an explicit tracking of what the revlog is about
Since the dawn of time, people have been forced to rely to lossy introspection
of the index filename to determine what the purpose and role of the revlog they
encounter is. This is hacky, error prone, inflexible, abstraction-leaky,
<insert-your-own-complaints-here>.
In f63299ee7e4d Raphaël introduced a new attribute to track this information:
`revlog_kind`. However it is initialized in an odd place and various instances
end up not having it set. In addition is only tracking some of the information
we end up having to introspect in various pieces of code.
So we add a new attribute that holds more data and is more strictly enforced.
This work is done in collaboration with Raphaël.
The `revlog_kind` one will be removed/adapted in the next changeset. We expect
to be able to clean up various existing piece of code and to simplify coming
work around the newer revlog format.
Differential Revision: https://phab.mercurial-scm.org/D10352
author | Pierre-Yves David <pierre-yves.david@octobus.net> |
---|---|
date | Tue, 06 Apr 2021 05:20:24 +0200 |
parents | d4ba4d51f85f |
children | df56e6bd37f6 2fe4efaa59af |
rev | line source |
---|---|
18635
fed06dd07665
worker: count the number of CPUs
Bryan O'Sullivan <bryano@fb.com>
parents:
diff
changeset
|
1 # worker.py - master-slave parallelism support |
fed06dd07665
worker: count the number of CPUs
Bryan O'Sullivan <bryano@fb.com>
parents:
diff
changeset
|
2 # |
fed06dd07665
worker: count the number of CPUs
Bryan O'Sullivan <bryano@fb.com>
parents:
diff
changeset
|
3 # Copyright 2013 Facebook, Inc. |
fed06dd07665
worker: count the number of CPUs
Bryan O'Sullivan <bryano@fb.com>
parents:
diff
changeset
|
4 # |
fed06dd07665
worker: count the number of CPUs
Bryan O'Sullivan <bryano@fb.com>
parents:
diff
changeset
|
5 # This software may be used and distributed according to the terms of the |
fed06dd07665
worker: count the number of CPUs
Bryan O'Sullivan <bryano@fb.com>
parents:
diff
changeset
|
6 # GNU General Public License version 2 or any later version. |
fed06dd07665
worker: count the number of CPUs
Bryan O'Sullivan <bryano@fb.com>
parents:
diff
changeset
|
7 |
25992
2d76f8a2d831
worker: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
25660
diff
changeset
|
8 from __future__ import absolute_import |
2d76f8a2d831
worker: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
25660
diff
changeset
|
9 |
2d76f8a2d831
worker: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
25660
diff
changeset
|
10 import errno |
2d76f8a2d831
worker: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
25660
diff
changeset
|
11 import os |
2d76f8a2d831
worker: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
25660
diff
changeset
|
12 import signal |
2d76f8a2d831
worker: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
25660
diff
changeset
|
13 import sys |
35427
02b36e860e0b
workers: implemented worker on windows
Wojciech Lis <wlis@fb.com>
parents:
34646
diff
changeset
|
14 import threading |
35432
86b8cc1f244e
worker: make windows workers daemons
Wojciech Lis <wlis@fb.com>
parents:
35431
diff
changeset
|
15 import time |
25992
2d76f8a2d831
worker: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
25660
diff
changeset
|
16 |
38729
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
17 try: |
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
18 import selectors |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
19 |
38729
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
20 selectors.BaseSelector |
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
21 except ImportError: |
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
22 from .thirdparty import selectors2 as selectors |
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
23 |
25992
2d76f8a2d831
worker: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
25660
diff
changeset
|
24 from .i18n import _ |
30396 | 25 from . import ( |
30635
a150173da1c1
py3: replace os.environ with encoding.environ (part 2 of 5)
Pulkit Goyal <7895pulkit@gmail.com>
parents:
30521
diff
changeset
|
26 encoding, |
30396 | 27 error, |
30639
d524c88511a7
py3: replace os.name with pycompat.osname (part 1 of 2)
Pulkit Goyal <7895pulkit@gmail.com>
parents:
30635
diff
changeset
|
28 pycompat, |
30521
86cd09bc13ba
worker: use os._exit for posix worker in all cases
Jun Wu <quark@fb.com>
parents:
30425
diff
changeset
|
29 scmutil, |
30396 | 30 util, |
31 ) | |
18635
fed06dd07665
worker: count the number of CPUs
Bryan O'Sullivan <bryano@fb.com>
parents:
diff
changeset
|
32 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
33 |
18635
fed06dd07665
worker: count the number of CPUs
Bryan O'Sullivan <bryano@fb.com>
parents:
diff
changeset
|
34 def countcpus(): |
fed06dd07665
worker: count the number of CPUs
Bryan O'Sullivan <bryano@fb.com>
parents:
diff
changeset
|
35 '''try to count the number of CPUs on the system''' |
26568
c0501c26b05c
worker: restore old countcpus code (issue4869)
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26063
diff
changeset
|
36 |
c0501c26b05c
worker: restore old countcpus code (issue4869)
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26063
diff
changeset
|
37 # posix |
18635
fed06dd07665
worker: count the number of CPUs
Bryan O'Sullivan <bryano@fb.com>
parents:
diff
changeset
|
38 try: |
43506
9f70512ae2cf
cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents:
43077
diff
changeset
|
39 n = int(os.sysconf('SC_NPROCESSORS_ONLN')) |
26568
c0501c26b05c
worker: restore old countcpus code (issue4869)
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26063
diff
changeset
|
40 if n > 0: |
c0501c26b05c
worker: restore old countcpus code (issue4869)
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26063
diff
changeset
|
41 return n |
c0501c26b05c
worker: restore old countcpus code (issue4869)
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26063
diff
changeset
|
42 except (AttributeError, ValueError): |
c0501c26b05c
worker: restore old countcpus code (issue4869)
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26063
diff
changeset
|
43 pass |
c0501c26b05c
worker: restore old countcpus code (issue4869)
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26063
diff
changeset
|
44 |
c0501c26b05c
worker: restore old countcpus code (issue4869)
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26063
diff
changeset
|
45 # windows |
c0501c26b05c
worker: restore old countcpus code (issue4869)
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26063
diff
changeset
|
46 try: |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
47 n = int(encoding.environ[b'NUMBER_OF_PROCESSORS']) |
26568
c0501c26b05c
worker: restore old countcpus code (issue4869)
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26063
diff
changeset
|
48 if n > 0: |
c0501c26b05c
worker: restore old countcpus code (issue4869)
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26063
diff
changeset
|
49 return n |
c0501c26b05c
worker: restore old countcpus code (issue4869)
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26063
diff
changeset
|
50 except (KeyError, ValueError): |
c0501c26b05c
worker: restore old countcpus code (issue4869)
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26063
diff
changeset
|
51 pass |
c0501c26b05c
worker: restore old countcpus code (issue4869)
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26063
diff
changeset
|
52 |
c0501c26b05c
worker: restore old countcpus code (issue4869)
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26063
diff
changeset
|
53 return 1 |
18636
dcb27c153a40
worker: estimate whether it's worth running a task in parallel
Bryan O'Sullivan <bryano@fb.com>
parents:
18635
diff
changeset
|
54 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
55 |
18636
dcb27c153a40
worker: estimate whether it's worth running a task in parallel
Bryan O'Sullivan <bryano@fb.com>
parents:
18635
diff
changeset
|
56 def _numworkers(ui): |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
57 s = ui.config(b'worker', b'numcpus') |
18636
dcb27c153a40
worker: estimate whether it's worth running a task in parallel
Bryan O'Sullivan <bryano@fb.com>
parents:
18635
diff
changeset
|
58 if s: |
dcb27c153a40
worker: estimate whether it's worth running a task in parallel
Bryan O'Sullivan <bryano@fb.com>
parents:
18635
diff
changeset
|
59 try: |
dcb27c153a40
worker: estimate whether it's worth running a task in parallel
Bryan O'Sullivan <bryano@fb.com>
parents:
18635
diff
changeset
|
60 n = int(s) |
dcb27c153a40
worker: estimate whether it's worth running a task in parallel
Bryan O'Sullivan <bryano@fb.com>
parents:
18635
diff
changeset
|
61 if n >= 1: |
dcb27c153a40
worker: estimate whether it's worth running a task in parallel
Bryan O'Sullivan <bryano@fb.com>
parents:
18635
diff
changeset
|
62 return n |
dcb27c153a40
worker: estimate whether it's worth running a task in parallel
Bryan O'Sullivan <bryano@fb.com>
parents:
18635
diff
changeset
|
63 except ValueError: |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
64 raise error.Abort(_(b'number of cpus must be an integer')) |
18636
dcb27c153a40
worker: estimate whether it's worth running a task in parallel
Bryan O'Sullivan <bryano@fb.com>
parents:
18635
diff
changeset
|
65 return min(max(countcpus(), 4), 32) |
dcb27c153a40
worker: estimate whether it's worth running a task in parallel
Bryan O'Sullivan <bryano@fb.com>
parents:
18635
diff
changeset
|
66 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
67 |
44248
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
68 if pycompat.ispy3: |
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
69 |
46240
a42502e9ae6d
worker: POSIX only supports workers from main thread (issue6460)
Joerg Sonnenberger <joerg@bec.de>
parents:
45942
diff
changeset
|
70 def ismainthread(): |
a42502e9ae6d
worker: POSIX only supports workers from main thread (issue6460)
Joerg Sonnenberger <joerg@bec.de>
parents:
45942
diff
changeset
|
71 return threading.current_thread() == threading.main_thread() |
a42502e9ae6d
worker: POSIX only supports workers from main thread (issue6460)
Joerg Sonnenberger <joerg@bec.de>
parents:
45942
diff
changeset
|
72 |
44248
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
73 class _blockingreader(object): |
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
74 def __init__(self, wrapped): |
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
75 self._wrapped = wrapped |
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
76 |
45390
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
45376
diff
changeset
|
77 # Do NOT implement readinto() by making it delegate to |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
45376
diff
changeset
|
78 # _wrapped.readinto(), since that is unbuffered. The unpickler is fine |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
45376
diff
changeset
|
79 # with just read() and readline(), so we don't need to implement it. |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
45376
diff
changeset
|
80 |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
45376
diff
changeset
|
81 def readline(self): |
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
45376
diff
changeset
|
82 return self._wrapped.readline() |
44248
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
83 |
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
84 # issue multiple reads until size is fulfilled |
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
85 def read(self, size=-1): |
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
86 if size < 0: |
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
87 return self._wrapped.readall() |
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
88 |
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
89 buf = bytearray(size) |
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
90 view = memoryview(buf) |
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
91 pos = 0 |
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
92 |
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
93 while pos < size: |
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
94 ret = self._wrapped.readinto(view[pos:]) |
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
95 if not ret: |
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
96 break |
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
97 pos += ret |
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
98 |
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
99 del view |
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
100 del buf[pos:] |
45390
7d24201b6447
worker: don't expose readinto() on _blockingreader since pickle is picky
Martin von Zweigbergk <martinvonz@google.com>
parents:
45376
diff
changeset
|
101 return bytes(buf) |
44248
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
102 |
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
103 |
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
104 else: |
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
105 |
46240
a42502e9ae6d
worker: POSIX only supports workers from main thread (issue6460)
Joerg Sonnenberger <joerg@bec.de>
parents:
45942
diff
changeset
|
106 def ismainthread(): |
46698
f64806207752
typing: disable a module-attr warning in the worker module's py2 code
Matt Harbison <matt_harbison@yahoo.com>
parents:
46240
diff
changeset
|
107 # pytype: disable=module-attr |
46240
a42502e9ae6d
worker: POSIX only supports workers from main thread (issue6460)
Joerg Sonnenberger <joerg@bec.de>
parents:
45942
diff
changeset
|
108 return isinstance(threading.current_thread(), threading._MainThread) |
46698
f64806207752
typing: disable a module-attr warning in the worker module's py2 code
Matt Harbison <matt_harbison@yahoo.com>
parents:
46240
diff
changeset
|
109 # pytype: enable=module-attr |
46240
a42502e9ae6d
worker: POSIX only supports workers from main thread (issue6460)
Joerg Sonnenberger <joerg@bec.de>
parents:
45942
diff
changeset
|
110 |
44248
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
111 def _blockingreader(wrapped): |
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
112 return wrapped |
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
113 |
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
114 |
35427
02b36e860e0b
workers: implemented worker on windows
Wojciech Lis <wlis@fb.com>
parents:
34646
diff
changeset
|
115 if pycompat.isposix or pycompat.iswindows: |
38730
69ed2cff4277
worker: rename variable to reflect constant
Gregory Szorc <gregory.szorc@gmail.com>
parents:
38729
diff
changeset
|
116 _STARTUP_COST = 0.01 |
38731
ef3838a47503
worker: ability to disable thread unsafe tasks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
38730
diff
changeset
|
117 # The Windows worker is thread based. If tasks are CPU bound, threads |
ef3838a47503
worker: ability to disable thread unsafe tasks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
38730
diff
changeset
|
118 # in the presence of the GIL result in excessive context switching and |
ef3838a47503
worker: ability to disable thread unsafe tasks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
38730
diff
changeset
|
119 # this overhead can slow down execution. |
ef3838a47503
worker: ability to disable thread unsafe tasks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
38730
diff
changeset
|
120 _DISALLOW_THREAD_UNSAFE = pycompat.iswindows |
18636
dcb27c153a40
worker: estimate whether it's worth running a task in parallel
Bryan O'Sullivan <bryano@fb.com>
parents:
18635
diff
changeset
|
121 else: |
38730
69ed2cff4277
worker: rename variable to reflect constant
Gregory Szorc <gregory.szorc@gmail.com>
parents:
38729
diff
changeset
|
122 _STARTUP_COST = 1e30 |
38731
ef3838a47503
worker: ability to disable thread unsafe tasks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
38730
diff
changeset
|
123 _DISALLOW_THREAD_UNSAFE = False |
18636
dcb27c153a40
worker: estimate whether it's worth running a task in parallel
Bryan O'Sullivan <bryano@fb.com>
parents:
18635
diff
changeset
|
124 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
125 |
38731
ef3838a47503
worker: ability to disable thread unsafe tasks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
38730
diff
changeset
|
126 def worthwhile(ui, costperop, nops, threadsafe=True): |
45942
89a2afe31e82
formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents:
45825
diff
changeset
|
127 """try to determine whether the benefit of multiple processes can |
89a2afe31e82
formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents:
45825
diff
changeset
|
128 outweigh the cost of starting them""" |
38731
ef3838a47503
worker: ability to disable thread unsafe tasks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
38730
diff
changeset
|
129 |
ef3838a47503
worker: ability to disable thread unsafe tasks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
38730
diff
changeset
|
130 if not threadsafe and _DISALLOW_THREAD_UNSAFE: |
ef3838a47503
worker: ability to disable thread unsafe tasks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
38730
diff
changeset
|
131 return False |
ef3838a47503
worker: ability to disable thread unsafe tasks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
38730
diff
changeset
|
132 |
18636
dcb27c153a40
worker: estimate whether it's worth running a task in parallel
Bryan O'Sullivan <bryano@fb.com>
parents:
18635
diff
changeset
|
133 linear = costperop * nops |
dcb27c153a40
worker: estimate whether it's worth running a task in parallel
Bryan O'Sullivan <bryano@fb.com>
parents:
18635
diff
changeset
|
134 workers = _numworkers(ui) |
38730
69ed2cff4277
worker: rename variable to reflect constant
Gregory Szorc <gregory.szorc@gmail.com>
parents:
38729
diff
changeset
|
135 benefit = linear - (_STARTUP_COST * workers + linear / workers) |
18636
dcb27c153a40
worker: estimate whether it's worth running a task in parallel
Bryan O'Sullivan <bryano@fb.com>
parents:
18635
diff
changeset
|
136 return benefit >= 0.15 |
18637
ac4dbceeb14a
worker: partition a list (of tasks) into equal-sized chunks
Bryan O'Sullivan <bryano@fb.com>
parents:
18636
diff
changeset
|
137 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
138 |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
139 def worker( |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
140 ui, costperarg, func, staticargs, args, hasretval=False, threadsafe=True |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
141 ): |
45942
89a2afe31e82
formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents:
45825
diff
changeset
|
142 """run a function, possibly in parallel in multiple worker |
18638
047110c0e2a8
worker: allow a function to be run in multiple worker processes
Bryan O'Sullivan <bryano@fb.com>
parents:
18637
diff
changeset
|
143 processes. |
047110c0e2a8
worker: allow a function to be run in multiple worker processes
Bryan O'Sullivan <bryano@fb.com>
parents:
18637
diff
changeset
|
144 |
047110c0e2a8
worker: allow a function to be run in multiple worker processes
Bryan O'Sullivan <bryano@fb.com>
parents:
18637
diff
changeset
|
145 returns a progress iterator |
047110c0e2a8
worker: allow a function to be run in multiple worker processes
Bryan O'Sullivan <bryano@fb.com>
parents:
18637
diff
changeset
|
146 |
047110c0e2a8
worker: allow a function to be run in multiple worker processes
Bryan O'Sullivan <bryano@fb.com>
parents:
18637
diff
changeset
|
147 costperarg - cost of a single task |
047110c0e2a8
worker: allow a function to be run in multiple worker processes
Bryan O'Sullivan <bryano@fb.com>
parents:
18637
diff
changeset
|
148 |
42455
5ca136bbd3f6
worker: support parallelization of functions with return values
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
40989
diff
changeset
|
149 func - function to run. It is expected to return a progress iterator. |
18638
047110c0e2a8
worker: allow a function to be run in multiple worker processes
Bryan O'Sullivan <bryano@fb.com>
parents:
18637
diff
changeset
|
150 |
047110c0e2a8
worker: allow a function to be run in multiple worker processes
Bryan O'Sullivan <bryano@fb.com>
parents:
18637
diff
changeset
|
151 staticargs - arguments to pass to every invocation of the function |
047110c0e2a8
worker: allow a function to be run in multiple worker processes
Bryan O'Sullivan <bryano@fb.com>
parents:
18637
diff
changeset
|
152 |
047110c0e2a8
worker: allow a function to be run in multiple worker processes
Bryan O'Sullivan <bryano@fb.com>
parents:
18637
diff
changeset
|
153 args - arguments to split into chunks, to pass to individual |
047110c0e2a8
worker: allow a function to be run in multiple worker processes
Bryan O'Sullivan <bryano@fb.com>
parents:
18637
diff
changeset
|
154 workers |
38731
ef3838a47503
worker: ability to disable thread unsafe tasks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
38730
diff
changeset
|
155 |
42455
5ca136bbd3f6
worker: support parallelization of functions with return values
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
40989
diff
changeset
|
156 hasretval - when True, func and the current function return an progress |
42522
d29db0a0c4eb
update: fix spurious unclean status bug shown by previous commit
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents:
42455
diff
changeset
|
157 iterator then a dict (encoded as an iterator that yield many (False, ..) |
d29db0a0c4eb
update: fix spurious unclean status bug shown by previous commit
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents:
42455
diff
changeset
|
158 then a (True, dict)). The dicts are joined in some arbitrary order, so |
d29db0a0c4eb
update: fix spurious unclean status bug shown by previous commit
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents:
42455
diff
changeset
|
159 overlapping keys are a bad idea. |
42455
5ca136bbd3f6
worker: support parallelization of functions with return values
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
40989
diff
changeset
|
160 |
38731
ef3838a47503
worker: ability to disable thread unsafe tasks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
38730
diff
changeset
|
161 threadsafe - whether work items are thread safe and can be executed using |
ef3838a47503
worker: ability to disable thread unsafe tasks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
38730
diff
changeset
|
162 a thread-based worker. Should be disabled for CPU heavy tasks that don't |
ef3838a47503
worker: ability to disable thread unsafe tasks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
38730
diff
changeset
|
163 release the GIL. |
45942
89a2afe31e82
formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents:
45825
diff
changeset
|
164 """ |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
165 enabled = ui.configbool(b'worker', b'enabled') |
46240
a42502e9ae6d
worker: POSIX only supports workers from main thread (issue6460)
Joerg Sonnenberger <joerg@bec.de>
parents:
45942
diff
changeset
|
166 if enabled and _platformworker is _posixworker and not ismainthread(): |
a42502e9ae6d
worker: POSIX only supports workers from main thread (issue6460)
Joerg Sonnenberger <joerg@bec.de>
parents:
45942
diff
changeset
|
167 # The POSIX worker has to install a handler for SIGCHLD. |
a42502e9ae6d
worker: POSIX only supports workers from main thread (issue6460)
Joerg Sonnenberger <joerg@bec.de>
parents:
45942
diff
changeset
|
168 # Python up to 3.9 only allows this in the main thread. |
a42502e9ae6d
worker: POSIX only supports workers from main thread (issue6460)
Joerg Sonnenberger <joerg@bec.de>
parents:
45942
diff
changeset
|
169 enabled = False |
a42502e9ae6d
worker: POSIX only supports workers from main thread (issue6460)
Joerg Sonnenberger <joerg@bec.de>
parents:
45942
diff
changeset
|
170 |
38731
ef3838a47503
worker: ability to disable thread unsafe tasks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
38730
diff
changeset
|
171 if enabled and worthwhile(ui, costperarg, len(args), threadsafe=threadsafe): |
42455
5ca136bbd3f6
worker: support parallelization of functions with return values
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
40989
diff
changeset
|
172 return _platformworker(ui, func, staticargs, args, hasretval) |
18638
047110c0e2a8
worker: allow a function to be run in multiple worker processes
Bryan O'Sullivan <bryano@fb.com>
parents:
18637
diff
changeset
|
173 return func(*staticargs + (args,)) |
047110c0e2a8
worker: allow a function to be run in multiple worker processes
Bryan O'Sullivan <bryano@fb.com>
parents:
18637
diff
changeset
|
174 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
175 |
42455
5ca136bbd3f6
worker: support parallelization of functions with return values
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
40989
diff
changeset
|
176 def _posixworker(ui, func, staticargs, args, hasretval): |
18638
047110c0e2a8
worker: allow a function to be run in multiple worker processes
Bryan O'Sullivan <bryano@fb.com>
parents:
18637
diff
changeset
|
177 workers = _numworkers(ui) |
18708
86524a70c0f6
worker: fix a race in SIGINT handling
Bryan O'Sullivan <bryano@fb.com>
parents:
18707
diff
changeset
|
178 oldhandler = signal.getsignal(signal.SIGINT) |
86524a70c0f6
worker: fix a race in SIGINT handling
Bryan O'Sullivan <bryano@fb.com>
parents:
18707
diff
changeset
|
179 signal.signal(signal.SIGINT, signal.SIG_IGN) |
30413 | 180 pids, problem = set(), [0] |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
181 |
30410
7a5d6e2fd2d5
worker: move killworkers and waitforworkers up
Jun Wu <quark@fb.com>
parents:
30396
diff
changeset
|
182 def killworkers(): |
30423
237b2883cbd8
worker: make sure killworkers() never be interrupted by another SIGCHLD
Yuya Nishihara <yuya@tcha.org>
parents:
30422
diff
changeset
|
183 # unregister SIGCHLD handler as all children will be killed. This |
237b2883cbd8
worker: make sure killworkers() never be interrupted by another SIGCHLD
Yuya Nishihara <yuya@tcha.org>
parents:
30422
diff
changeset
|
184 # function shouldn't be interrupted by another SIGCHLD; otherwise pids |
237b2883cbd8
worker: make sure killworkers() never be interrupted by another SIGCHLD
Yuya Nishihara <yuya@tcha.org>
parents:
30422
diff
changeset
|
185 # could be updated while iterating, which would cause inconsistency. |
237b2883cbd8
worker: make sure killworkers() never be interrupted by another SIGCHLD
Yuya Nishihara <yuya@tcha.org>
parents:
30422
diff
changeset
|
186 signal.signal(signal.SIGCHLD, oldchldhandler) |
30410
7a5d6e2fd2d5
worker: move killworkers and waitforworkers up
Jun Wu <quark@fb.com>
parents:
30396
diff
changeset
|
187 # if one worker bails, there's no good reason to wait for the rest |
7a5d6e2fd2d5
worker: move killworkers and waitforworkers up
Jun Wu <quark@fb.com>
parents:
30396
diff
changeset
|
188 for p in pids: |
7a5d6e2fd2d5
worker: move killworkers and waitforworkers up
Jun Wu <quark@fb.com>
parents:
30396
diff
changeset
|
189 try: |
7a5d6e2fd2d5
worker: move killworkers and waitforworkers up
Jun Wu <quark@fb.com>
parents:
30396
diff
changeset
|
190 os.kill(p, signal.SIGTERM) |
7a5d6e2fd2d5
worker: move killworkers and waitforworkers up
Jun Wu <quark@fb.com>
parents:
30396
diff
changeset
|
191 except OSError as err: |
7a5d6e2fd2d5
worker: move killworkers and waitforworkers up
Jun Wu <quark@fb.com>
parents:
30396
diff
changeset
|
192 if err.errno != errno.ESRCH: |
7a5d6e2fd2d5
worker: move killworkers and waitforworkers up
Jun Wu <quark@fb.com>
parents:
30396
diff
changeset
|
193 raise |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
194 |
30412
7bc25549e084
worker: allow waitforworkers to be non-blocking
Jun Wu <quark@fb.com>
parents:
30411
diff
changeset
|
195 def waitforworkers(blocking=True): |
30414
5069a8a40b1b
worker: make waitforworkers reentrant
Jun Wu <quark@fb.com>
parents:
30413
diff
changeset
|
196 for pid in pids.copy(): |
5069a8a40b1b
worker: make waitforworkers reentrant
Jun Wu <quark@fb.com>
parents:
30413
diff
changeset
|
197 p = st = 0 |
5069a8a40b1b
worker: make waitforworkers reentrant
Jun Wu <quark@fb.com>
parents:
30413
diff
changeset
|
198 while True: |
5069a8a40b1b
worker: make waitforworkers reentrant
Jun Wu <quark@fb.com>
parents:
30413
diff
changeset
|
199 try: |
5069a8a40b1b
worker: make waitforworkers reentrant
Jun Wu <quark@fb.com>
parents:
30413
diff
changeset
|
200 p, st = os.waitpid(pid, (0 if blocking else os.WNOHANG)) |
30422
0e6ce6313e47
worker: fix missed break on successful waitpid()
Yuya Nishihara <yuya@tcha.org>
parents:
30416
diff
changeset
|
201 break |
30414
5069a8a40b1b
worker: make waitforworkers reentrant
Jun Wu <quark@fb.com>
parents:
30413
diff
changeset
|
202 except OSError as e: |
5069a8a40b1b
worker: make waitforworkers reentrant
Jun Wu <quark@fb.com>
parents:
30413
diff
changeset
|
203 if e.errno == errno.EINTR: |
5069a8a40b1b
worker: make waitforworkers reentrant
Jun Wu <quark@fb.com>
parents:
30413
diff
changeset
|
204 continue |
5069a8a40b1b
worker: make waitforworkers reentrant
Jun Wu <quark@fb.com>
parents:
30413
diff
changeset
|
205 elif e.errno == errno.ECHILD: |
30425
03f7aa2bd0e3
worker: discard waited pid by anyone who noticed it first
Yuya Nishihara <yuya@tcha.org>
parents:
30424
diff
changeset
|
206 # child would already be reaped, but pids yet been |
03f7aa2bd0e3
worker: discard waited pid by anyone who noticed it first
Yuya Nishihara <yuya@tcha.org>
parents:
30424
diff
changeset
|
207 # updated (maybe interrupted just after waitpid) |
03f7aa2bd0e3
worker: discard waited pid by anyone who noticed it first
Yuya Nishihara <yuya@tcha.org>
parents:
30424
diff
changeset
|
208 pids.discard(pid) |
03f7aa2bd0e3
worker: discard waited pid by anyone who noticed it first
Yuya Nishihara <yuya@tcha.org>
parents:
30424
diff
changeset
|
209 break |
30414
5069a8a40b1b
worker: make waitforworkers reentrant
Jun Wu <quark@fb.com>
parents:
30413
diff
changeset
|
210 else: |
5069a8a40b1b
worker: make waitforworkers reentrant
Jun Wu <quark@fb.com>
parents:
30413
diff
changeset
|
211 raise |
31063
18fb3cf572b4
worker: ignore meaningless exit status indication returned by os.waitpid()
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents:
30639
diff
changeset
|
212 if not p: |
18fb3cf572b4
worker: ignore meaningless exit status indication returned by os.waitpid()
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents:
30639
diff
changeset
|
213 # skip subsequent steps, because child process should |
18fb3cf572b4
worker: ignore meaningless exit status indication returned by os.waitpid()
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents:
30639
diff
changeset
|
214 # be still running in this case |
18fb3cf572b4
worker: ignore meaningless exit status indication returned by os.waitpid()
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents:
30639
diff
changeset
|
215 continue |
18fb3cf572b4
worker: ignore meaningless exit status indication returned by os.waitpid()
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents:
30639
diff
changeset
|
216 pids.discard(p) |
18fb3cf572b4
worker: ignore meaningless exit status indication returned by os.waitpid()
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents:
30639
diff
changeset
|
217 st = _exitstatus(st) |
30410
7a5d6e2fd2d5
worker: move killworkers and waitforworkers up
Jun Wu <quark@fb.com>
parents:
30396
diff
changeset
|
218 if st and not problem[0]: |
7a5d6e2fd2d5
worker: move killworkers and waitforworkers up
Jun Wu <quark@fb.com>
parents:
30396
diff
changeset
|
219 problem[0] = st |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
220 |
30415
e8fb03cfbbde
worker: add a SIGCHLD handler to collect worker immediately
Jun Wu <quark@fb.com>
parents:
30414
diff
changeset
|
221 def sigchldhandler(signum, frame): |
e8fb03cfbbde
worker: add a SIGCHLD handler to collect worker immediately
Jun Wu <quark@fb.com>
parents:
30414
diff
changeset
|
222 waitforworkers(blocking=False) |
30424
f2d13eb85198
worker: kill workers after all zombie processes are reaped
Yuya Nishihara <yuya@tcha.org>
parents:
30423
diff
changeset
|
223 if problem[0]: |
f2d13eb85198
worker: kill workers after all zombie processes are reaped
Yuya Nishihara <yuya@tcha.org>
parents:
30423
diff
changeset
|
224 killworkers() |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
225 |
30415
e8fb03cfbbde
worker: add a SIGCHLD handler to collect worker immediately
Jun Wu <quark@fb.com>
parents:
30414
diff
changeset
|
226 oldchldhandler = signal.signal(signal.SIGCHLD, sigchldhandler) |
31696
9d3d56aa1a9f
worker: flush ui buffers before running the worker
David Soria Parra <davidsp@fb.com>
parents:
31119
diff
changeset
|
227 ui.flush() |
32112
31763785094b
worker: rewrite error handling so os._exit covers all cases
Jun Wu <quark@fb.com>
parents:
32043
diff
changeset
|
228 parentpid = os.getpid() |
38729
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
229 pipes = [] |
42522
d29db0a0c4eb
update: fix spurious unclean status bug shown by previous commit
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents:
42455
diff
changeset
|
230 retval = {} |
45376
26eb62bd0550
posixworker: avoid creating workers that end up getting no work
Martin von Zweigbergk <martinvonz@google.com>
parents:
44248
diff
changeset
|
231 for pargs in partition(args, min(workers, len(args))): |
38729
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
232 # Every worker gets its own pipe to send results on, so we don't have to |
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
233 # implement atomic writes larger than PIPE_BUF. Each forked process has |
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
234 # its own pipe's descriptors in the local variables, and the parent |
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
235 # process has the full list of pipe descriptors (and it doesn't really |
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
236 # care what order they're in). |
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
237 rfd, wfd = os.pipe() |
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
238 pipes.append((rfd, wfd)) |
32112
31763785094b
worker: rewrite error handling so os._exit covers all cases
Jun Wu <quark@fb.com>
parents:
32043
diff
changeset
|
239 # make sure we use os._exit in all worker code paths. otherwise the |
31763785094b
worker: rewrite error handling so os._exit covers all cases
Jun Wu <quark@fb.com>
parents:
32043
diff
changeset
|
240 # worker may do some clean-ups which could cause surprises like |
31763785094b
worker: rewrite error handling so os._exit covers all cases
Jun Wu <quark@fb.com>
parents:
32043
diff
changeset
|
241 # deadlock. see sshpeer.cleanup for example. |
31763785094b
worker: rewrite error handling so os._exit covers all cases
Jun Wu <quark@fb.com>
parents:
32043
diff
changeset
|
242 # override error handling *before* fork. this is necessary because |
31763785094b
worker: rewrite error handling so os._exit covers all cases
Jun Wu <quark@fb.com>
parents:
32043
diff
changeset
|
243 # exception (signal) may arrive after fork, before "pid =" assignment |
31763785094b
worker: rewrite error handling so os._exit covers all cases
Jun Wu <quark@fb.com>
parents:
32043
diff
changeset
|
244 # completes, and other exception handler (dispatch.py) can lead to |
31763785094b
worker: rewrite error handling so os._exit covers all cases
Jun Wu <quark@fb.com>
parents:
32043
diff
changeset
|
245 # unexpected code path without os._exit. |
31763785094b
worker: rewrite error handling so os._exit covers all cases
Jun Wu <quark@fb.com>
parents:
32043
diff
changeset
|
246 ret = -1 |
31763785094b
worker: rewrite error handling so os._exit covers all cases
Jun Wu <quark@fb.com>
parents:
32043
diff
changeset
|
247 try: |
31763785094b
worker: rewrite error handling so os._exit covers all cases
Jun Wu <quark@fb.com>
parents:
32043
diff
changeset
|
248 pid = os.fork() |
31763785094b
worker: rewrite error handling so os._exit covers all cases
Jun Wu <quark@fb.com>
parents:
32043
diff
changeset
|
249 if pid == 0: |
31763785094b
worker: rewrite error handling so os._exit covers all cases
Jun Wu <quark@fb.com>
parents:
32043
diff
changeset
|
250 signal.signal(signal.SIGINT, oldhandler) |
31763785094b
worker: rewrite error handling so os._exit covers all cases
Jun Wu <quark@fb.com>
parents:
32043
diff
changeset
|
251 signal.signal(signal.SIGCHLD, oldchldhandler) |
30521
86cd09bc13ba
worker: use os._exit for posix worker in all cases
Jun Wu <quark@fb.com>
parents:
30425
diff
changeset
|
252 |
32112
31763785094b
worker: rewrite error handling so os._exit covers all cases
Jun Wu <quark@fb.com>
parents:
32043
diff
changeset
|
253 def workerfunc(): |
38729
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
254 for r, w in pipes[:-1]: |
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
255 os.close(r) |
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
256 os.close(w) |
32112
31763785094b
worker: rewrite error handling so os._exit covers all cases
Jun Wu <quark@fb.com>
parents:
32043
diff
changeset
|
257 os.close(rfd) |
38535
8c38d2948217
worker: support more return types in posix worker
Danny Hooper <hooper@google.com>
parents:
37844
diff
changeset
|
258 for result in func(*(staticargs + (pargs,))): |
42522
d29db0a0c4eb
update: fix spurious unclean status bug shown by previous commit
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents:
42455
diff
changeset
|
259 os.write(wfd, util.pickle.dumps(result)) |
32112
31763785094b
worker: rewrite error handling so os._exit covers all cases
Jun Wu <quark@fb.com>
parents:
32043
diff
changeset
|
260 return 0 |
31763785094b
worker: rewrite error handling so os._exit covers all cases
Jun Wu <quark@fb.com>
parents:
32043
diff
changeset
|
261 |
31763785094b
worker: rewrite error handling so os._exit covers all cases
Jun Wu <quark@fb.com>
parents:
32043
diff
changeset
|
262 ret = scmutil.callcatch(ui, workerfunc) |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
263 except: # parent re-raises, child never returns |
32112
31763785094b
worker: rewrite error handling so os._exit covers all cases
Jun Wu <quark@fb.com>
parents:
32043
diff
changeset
|
264 if os.getpid() == parentpid: |
31763785094b
worker: rewrite error handling so os._exit covers all cases
Jun Wu <quark@fb.com>
parents:
32043
diff
changeset
|
265 raise |
31763785094b
worker: rewrite error handling so os._exit covers all cases
Jun Wu <quark@fb.com>
parents:
32043
diff
changeset
|
266 exctype = sys.exc_info()[0] |
31763785094b
worker: rewrite error handling so os._exit covers all cases
Jun Wu <quark@fb.com>
parents:
32043
diff
changeset
|
267 force = not issubclass(exctype, KeyboardInterrupt) |
31763785094b
worker: rewrite error handling so os._exit covers all cases
Jun Wu <quark@fb.com>
parents:
32043
diff
changeset
|
268 ui.traceback(force=force) |
31763785094b
worker: rewrite error handling so os._exit covers all cases
Jun Wu <quark@fb.com>
parents:
32043
diff
changeset
|
269 finally: |
31763785094b
worker: rewrite error handling so os._exit covers all cases
Jun Wu <quark@fb.com>
parents:
32043
diff
changeset
|
270 if os.getpid() != parentpid: |
31118
a91c62752d08
worker: flush messages written by child processes before exit
Yuya Nishihara <yuya@tcha.org>
parents:
31063
diff
changeset
|
271 try: |
a91c62752d08
worker: flush messages written by child processes before exit
Yuya Nishihara <yuya@tcha.org>
parents:
31063
diff
changeset
|
272 ui.flush() |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
273 except: # never returns, no re-raises |
32112
31763785094b
worker: rewrite error handling so os._exit covers all cases
Jun Wu <quark@fb.com>
parents:
32043
diff
changeset
|
274 pass |
30521
86cd09bc13ba
worker: use os._exit for posix worker in all cases
Jun Wu <quark@fb.com>
parents:
30425
diff
changeset
|
275 finally: |
32112
31763785094b
worker: rewrite error handling so os._exit covers all cases
Jun Wu <quark@fb.com>
parents:
32043
diff
changeset
|
276 os._exit(ret & 255) |
30413 | 277 pids.add(pid) |
38729
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
278 selector = selectors.DefaultSelector() |
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
279 for rfd, wfd in pipes: |
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
280 os.close(wfd) |
44248
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
281 selector.register(os.fdopen(rfd, 'rb', 0), selectors.EVENT_READ) |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
282 |
18638
047110c0e2a8
worker: allow a function to be run in multiple worker processes
Bryan O'Sullivan <bryano@fb.com>
parents:
18637
diff
changeset
|
283 def cleanup(): |
047110c0e2a8
worker: allow a function to be run in multiple worker processes
Bryan O'Sullivan <bryano@fb.com>
parents:
18637
diff
changeset
|
284 signal.signal(signal.SIGINT, oldhandler) |
30416
c27614f2dec1
worker: stop using a separate thread waiting for children
Jun Wu <quark@fb.com>
parents:
30415
diff
changeset
|
285 waitforworkers() |
30415
e8fb03cfbbde
worker: add a SIGCHLD handler to collect worker immediately
Jun Wu <quark@fb.com>
parents:
30414
diff
changeset
|
286 signal.signal(signal.SIGCHLD, oldchldhandler) |
38740
c08ea1e219c0
worker: call selector.close() to release polling resources
Yuya Nishihara <yuya@tcha.org>
parents:
38731
diff
changeset
|
287 selector.close() |
40988
03f7d0822ec1
worker: do not swallow exception occurred in main process
Yuya Nishihara <yuya@tcha.org>
parents:
38740
diff
changeset
|
288 return problem[0] |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
289 |
18638
047110c0e2a8
worker: allow a function to be run in multiple worker processes
Bryan O'Sullivan <bryano@fb.com>
parents:
18637
diff
changeset
|
290 try: |
38729
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
291 openpipes = len(pipes) |
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
292 while openpipes > 0: |
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
293 for key, events in selector.select(): |
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
294 try: |
44248
12491abf93bd
worker: manually buffer reads from pickle stream
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
parents:
44226
diff
changeset
|
295 res = util.pickle.load(_blockingreader(key.fileobj)) |
42455
5ca136bbd3f6
worker: support parallelization of functions with return values
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
40989
diff
changeset
|
296 if hasretval and res[0]: |
42522
d29db0a0c4eb
update: fix spurious unclean status bug shown by previous commit
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents:
42455
diff
changeset
|
297 retval.update(res[1]) |
42455
5ca136bbd3f6
worker: support parallelization of functions with return values
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
40989
diff
changeset
|
298 else: |
5ca136bbd3f6
worker: support parallelization of functions with return values
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
40989
diff
changeset
|
299 yield res |
38729
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
300 except EOFError: |
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
301 selector.unregister(key.fileobj) |
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
302 key.fileobj.close() |
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
303 openpipes -= 1 |
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
304 except IOError as e: |
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
305 if e.errno == errno.EINTR: |
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
306 continue |
9e6afe7fca31
worker: use one pipe per posix worker and select() in parent process
Danny Hooper <hooper@google.com>
parents:
38535
diff
changeset
|
307 raise |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
308 except: # re-raises |
18709
9955fc5ee24b
worker: handle worker failures more aggressively
Bryan O'Sullivan <bryano@fb.com>
parents:
18708
diff
changeset
|
309 killworkers() |
18638
047110c0e2a8
worker: allow a function to be run in multiple worker processes
Bryan O'Sullivan <bryano@fb.com>
parents:
18637
diff
changeset
|
310 cleanup() |
047110c0e2a8
worker: allow a function to be run in multiple worker processes
Bryan O'Sullivan <bryano@fb.com>
parents:
18637
diff
changeset
|
311 raise |
40988
03f7d0822ec1
worker: do not swallow exception occurred in main process
Yuya Nishihara <yuya@tcha.org>
parents:
38740
diff
changeset
|
312 status = cleanup() |
03f7d0822ec1
worker: do not swallow exception occurred in main process
Yuya Nishihara <yuya@tcha.org>
parents:
38740
diff
changeset
|
313 if status: |
03f7d0822ec1
worker: do not swallow exception occurred in main process
Yuya Nishihara <yuya@tcha.org>
parents:
38740
diff
changeset
|
314 if status < 0: |
03f7d0822ec1
worker: do not swallow exception occurred in main process
Yuya Nishihara <yuya@tcha.org>
parents:
38740
diff
changeset
|
315 os.kill(os.getpid(), -status) |
45825
8f07f5a9c3de
worker: raise exception instead of calling sys.exit() with child's code
Martin von Zweigbergk <martinvonz@google.com>
parents:
45390
diff
changeset
|
316 raise error.WorkerError(status) |
42455
5ca136bbd3f6
worker: support parallelization of functions with return values
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
40989
diff
changeset
|
317 if hasretval: |
42522
d29db0a0c4eb
update: fix spurious unclean status bug shown by previous commit
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents:
42455
diff
changeset
|
318 yield True, retval |
18638
047110c0e2a8
worker: allow a function to be run in multiple worker processes
Bryan O'Sullivan <bryano@fb.com>
parents:
18637
diff
changeset
|
319 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
320 |
18707
d1a2b086d058
worker: on error, exit similarly to the first failing worker
Bryan O'Sullivan <bryano@fb.com>
parents:
18638
diff
changeset
|
321 def _posixexitstatus(code): |
45942
89a2afe31e82
formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents:
45825
diff
changeset
|
322 """convert a posix exit status into the same form returned by |
18707
d1a2b086d058
worker: on error, exit similarly to the first failing worker
Bryan O'Sullivan <bryano@fb.com>
parents:
18638
diff
changeset
|
323 os.spawnv |
d1a2b086d058
worker: on error, exit similarly to the first failing worker
Bryan O'Sullivan <bryano@fb.com>
parents:
18638
diff
changeset
|
324 |
45942
89a2afe31e82
formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents:
45825
diff
changeset
|
325 returns None if the process was stopped instead of exiting""" |
18707
d1a2b086d058
worker: on error, exit similarly to the first failing worker
Bryan O'Sullivan <bryano@fb.com>
parents:
18638
diff
changeset
|
326 if os.WIFEXITED(code): |
d1a2b086d058
worker: on error, exit similarly to the first failing worker
Bryan O'Sullivan <bryano@fb.com>
parents:
18638
diff
changeset
|
327 return os.WEXITSTATUS(code) |
d1a2b086d058
worker: on error, exit similarly to the first failing worker
Bryan O'Sullivan <bryano@fb.com>
parents:
18638
diff
changeset
|
328 elif os.WIFSIGNALED(code): |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
329 return -(os.WTERMSIG(code)) |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
330 |
18707
d1a2b086d058
worker: on error, exit similarly to the first failing worker
Bryan O'Sullivan <bryano@fb.com>
parents:
18638
diff
changeset
|
331 |
42455
5ca136bbd3f6
worker: support parallelization of functions with return values
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
40989
diff
changeset
|
332 def _windowsworker(ui, func, staticargs, args, hasretval): |
35427
02b36e860e0b
workers: implemented worker on windows
Wojciech Lis <wlis@fb.com>
parents:
34646
diff
changeset
|
333 class Worker(threading.Thread): |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
334 def __init__( |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
335 self, taskqueue, resultqueue, func, staticargs, *args, **kwargs |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
336 ): |
40443
909c31805f54
py3: roll up threading.Thread constructor args into **kwargs
Matt Harbison <matt_harbison@yahoo.com>
parents:
38740
diff
changeset
|
337 threading.Thread.__init__(self, *args, **kwargs) |
35427
02b36e860e0b
workers: implemented worker on windows
Wojciech Lis <wlis@fb.com>
parents:
34646
diff
changeset
|
338 self._taskqueue = taskqueue |
02b36e860e0b
workers: implemented worker on windows
Wojciech Lis <wlis@fb.com>
parents:
34646
diff
changeset
|
339 self._resultqueue = resultqueue |
02b36e860e0b
workers: implemented worker on windows
Wojciech Lis <wlis@fb.com>
parents:
34646
diff
changeset
|
340 self._func = func |
02b36e860e0b
workers: implemented worker on windows
Wojciech Lis <wlis@fb.com>
parents:
34646
diff
changeset
|
341 self._staticargs = staticargs |
35428
71427ff1dff8
workers: handling exceptions in windows workers
Wojciech Lis <wlis@fb.com>
parents:
35427
diff
changeset
|
342 self._interrupted = False |
35432
86b8cc1f244e
worker: make windows workers daemons
Wojciech Lis <wlis@fb.com>
parents:
35431
diff
changeset
|
343 self.daemon = True |
35428
71427ff1dff8
workers: handling exceptions in windows workers
Wojciech Lis <wlis@fb.com>
parents:
35427
diff
changeset
|
344 self.exception = None |
71427ff1dff8
workers: handling exceptions in windows workers
Wojciech Lis <wlis@fb.com>
parents:
35427
diff
changeset
|
345 |
71427ff1dff8
workers: handling exceptions in windows workers
Wojciech Lis <wlis@fb.com>
parents:
35427
diff
changeset
|
346 def interrupt(self): |
71427ff1dff8
workers: handling exceptions in windows workers
Wojciech Lis <wlis@fb.com>
parents:
35427
diff
changeset
|
347 self._interrupted = True |
35427
02b36e860e0b
workers: implemented worker on windows
Wojciech Lis <wlis@fb.com>
parents:
34646
diff
changeset
|
348 |
02b36e860e0b
workers: implemented worker on windows
Wojciech Lis <wlis@fb.com>
parents:
34646
diff
changeset
|
349 def run(self): |
35428
71427ff1dff8
workers: handling exceptions in windows workers
Wojciech Lis <wlis@fb.com>
parents:
35427
diff
changeset
|
350 try: |
71427ff1dff8
workers: handling exceptions in windows workers
Wojciech Lis <wlis@fb.com>
parents:
35427
diff
changeset
|
351 while not self._taskqueue.empty(): |
71427ff1dff8
workers: handling exceptions in windows workers
Wojciech Lis <wlis@fb.com>
parents:
35427
diff
changeset
|
352 try: |
42522
d29db0a0c4eb
update: fix spurious unclean status bug shown by previous commit
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents:
42455
diff
changeset
|
353 args = self._taskqueue.get_nowait() |
35428
71427ff1dff8
workers: handling exceptions in windows workers
Wojciech Lis <wlis@fb.com>
parents:
35427
diff
changeset
|
354 for res in self._func(*self._staticargs + (args,)): |
42522
d29db0a0c4eb
update: fix spurious unclean status bug shown by previous commit
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents:
42455
diff
changeset
|
355 self._resultqueue.put(res) |
35428
71427ff1dff8
workers: handling exceptions in windows workers
Wojciech Lis <wlis@fb.com>
parents:
35427
diff
changeset
|
356 # threading doesn't provide a native way to |
71427ff1dff8
workers: handling exceptions in windows workers
Wojciech Lis <wlis@fb.com>
parents:
35427
diff
changeset
|
357 # interrupt execution. handle it manually at every |
71427ff1dff8
workers: handling exceptions in windows workers
Wojciech Lis <wlis@fb.com>
parents:
35427
diff
changeset
|
358 # iteration. |
71427ff1dff8
workers: handling exceptions in windows workers
Wojciech Lis <wlis@fb.com>
parents:
35427
diff
changeset
|
359 if self._interrupted: |
71427ff1dff8
workers: handling exceptions in windows workers
Wojciech Lis <wlis@fb.com>
parents:
35427
diff
changeset
|
360 return |
37844
8fb9985382be
pycompat: export queue module instead of symbols in module (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents:
36835
diff
changeset
|
361 except pycompat.queue.Empty: |
35428
71427ff1dff8
workers: handling exceptions in windows workers
Wojciech Lis <wlis@fb.com>
parents:
35427
diff
changeset
|
362 break |
71427ff1dff8
workers: handling exceptions in windows workers
Wojciech Lis <wlis@fb.com>
parents:
35427
diff
changeset
|
363 except Exception as e: |
71427ff1dff8
workers: handling exceptions in windows workers
Wojciech Lis <wlis@fb.com>
parents:
35427
diff
changeset
|
364 # store the exception such that the main thread can resurface |
71427ff1dff8
workers: handling exceptions in windows workers
Wojciech Lis <wlis@fb.com>
parents:
35427
diff
changeset
|
365 # it as if the func was running without workers. |
71427ff1dff8
workers: handling exceptions in windows workers
Wojciech Lis <wlis@fb.com>
parents:
35427
diff
changeset
|
366 self.exception = e |
71427ff1dff8
workers: handling exceptions in windows workers
Wojciech Lis <wlis@fb.com>
parents:
35427
diff
changeset
|
367 raise |
71427ff1dff8
workers: handling exceptions in windows workers
Wojciech Lis <wlis@fb.com>
parents:
35427
diff
changeset
|
368 |
71427ff1dff8
workers: handling exceptions in windows workers
Wojciech Lis <wlis@fb.com>
parents:
35427
diff
changeset
|
369 threads = [] |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
370 |
35432
86b8cc1f244e
worker: make windows workers daemons
Wojciech Lis <wlis@fb.com>
parents:
35431
diff
changeset
|
371 def trykillworkers(): |
86b8cc1f244e
worker: make windows workers daemons
Wojciech Lis <wlis@fb.com>
parents:
35431
diff
changeset
|
372 # Allow up to 1 second to clean worker threads nicely |
86b8cc1f244e
worker: make windows workers daemons
Wojciech Lis <wlis@fb.com>
parents:
35431
diff
changeset
|
373 cleanupend = time.time() + 1 |
35428
71427ff1dff8
workers: handling exceptions in windows workers
Wojciech Lis <wlis@fb.com>
parents:
35427
diff
changeset
|
374 for t in threads: |
71427ff1dff8
workers: handling exceptions in windows workers
Wojciech Lis <wlis@fb.com>
parents:
35427
diff
changeset
|
375 t.interrupt() |
71427ff1dff8
workers: handling exceptions in windows workers
Wojciech Lis <wlis@fb.com>
parents:
35427
diff
changeset
|
376 for t in threads: |
35432
86b8cc1f244e
worker: make windows workers daemons
Wojciech Lis <wlis@fb.com>
parents:
35431
diff
changeset
|
377 remainingtime = cleanupend - time.time() |
86b8cc1f244e
worker: make windows workers daemons
Wojciech Lis <wlis@fb.com>
parents:
35431
diff
changeset
|
378 t.join(remainingtime) |
35428
71427ff1dff8
workers: handling exceptions in windows workers
Wojciech Lis <wlis@fb.com>
parents:
35427
diff
changeset
|
379 if t.is_alive(): |
35432
86b8cc1f244e
worker: make windows workers daemons
Wojciech Lis <wlis@fb.com>
parents:
35431
diff
changeset
|
380 # pass over the workers joining failure. it is more |
86b8cc1f244e
worker: make windows workers daemons
Wojciech Lis <wlis@fb.com>
parents:
35431
diff
changeset
|
381 # important to surface the inital exception than the |
86b8cc1f244e
worker: make windows workers daemons
Wojciech Lis <wlis@fb.com>
parents:
35431
diff
changeset
|
382 # fact that one of workers may be processing a large |
86b8cc1f244e
worker: make windows workers daemons
Wojciech Lis <wlis@fb.com>
parents:
35431
diff
changeset
|
383 # task and does not get to handle the interruption. |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
384 ui.warn( |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
385 _( |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
386 b"failed to kill worker threads while " |
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
387 b"handling an exception\n" |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
388 ) |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
389 ) |
35432
86b8cc1f244e
worker: make windows workers daemons
Wojciech Lis <wlis@fb.com>
parents:
35431
diff
changeset
|
390 return |
35427
02b36e860e0b
workers: implemented worker on windows
Wojciech Lis <wlis@fb.com>
parents:
34646
diff
changeset
|
391 |
02b36e860e0b
workers: implemented worker on windows
Wojciech Lis <wlis@fb.com>
parents:
34646
diff
changeset
|
392 workers = _numworkers(ui) |
37844
8fb9985382be
pycompat: export queue module instead of symbols in module (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents:
36835
diff
changeset
|
393 resultqueue = pycompat.queue.Queue() |
8fb9985382be
pycompat: export queue module instead of symbols in module (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents:
36835
diff
changeset
|
394 taskqueue = pycompat.queue.Queue() |
42522
d29db0a0c4eb
update: fix spurious unclean status bug shown by previous commit
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents:
42455
diff
changeset
|
395 retval = {} |
35427
02b36e860e0b
workers: implemented worker on windows
Wojciech Lis <wlis@fb.com>
parents:
34646
diff
changeset
|
396 # partition work to more pieces than workers to minimize the chance |
02b36e860e0b
workers: implemented worker on windows
Wojciech Lis <wlis@fb.com>
parents:
34646
diff
changeset
|
397 # of uneven distribution of large tasks between the workers |
42522
d29db0a0c4eb
update: fix spurious unclean status bug shown by previous commit
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents:
42455
diff
changeset
|
398 for pargs in partition(args, workers * 20): |
35427
02b36e860e0b
workers: implemented worker on windows
Wojciech Lis <wlis@fb.com>
parents:
34646
diff
changeset
|
399 taskqueue.put(pargs) |
02b36e860e0b
workers: implemented worker on windows
Wojciech Lis <wlis@fb.com>
parents:
34646
diff
changeset
|
400 for _i in range(workers): |
02b36e860e0b
workers: implemented worker on windows
Wojciech Lis <wlis@fb.com>
parents:
34646
diff
changeset
|
401 t = Worker(taskqueue, resultqueue, func, staticargs) |
02b36e860e0b
workers: implemented worker on windows
Wojciech Lis <wlis@fb.com>
parents:
34646
diff
changeset
|
402 threads.append(t) |
02b36e860e0b
workers: implemented worker on windows
Wojciech Lis <wlis@fb.com>
parents:
34646
diff
changeset
|
403 t.start() |
35432
86b8cc1f244e
worker: make windows workers daemons
Wojciech Lis <wlis@fb.com>
parents:
35431
diff
changeset
|
404 try: |
86b8cc1f244e
worker: make windows workers daemons
Wojciech Lis <wlis@fb.com>
parents:
35431
diff
changeset
|
405 while len(threads) > 0: |
86b8cc1f244e
worker: make windows workers daemons
Wojciech Lis <wlis@fb.com>
parents:
35431
diff
changeset
|
406 while not resultqueue.empty(): |
42522
d29db0a0c4eb
update: fix spurious unclean status bug shown by previous commit
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents:
42455
diff
changeset
|
407 res = resultqueue.get() |
42455
5ca136bbd3f6
worker: support parallelization of functions with return values
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
40989
diff
changeset
|
408 if hasretval and res[0]: |
42522
d29db0a0c4eb
update: fix spurious unclean status bug shown by previous commit
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents:
42455
diff
changeset
|
409 retval.update(res[1]) |
42455
5ca136bbd3f6
worker: support parallelization of functions with return values
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
40989
diff
changeset
|
410 else: |
5ca136bbd3f6
worker: support parallelization of functions with return values
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
40989
diff
changeset
|
411 yield res |
35432
86b8cc1f244e
worker: make windows workers daemons
Wojciech Lis <wlis@fb.com>
parents:
35431
diff
changeset
|
412 threads[0].join(0.05) |
86b8cc1f244e
worker: make windows workers daemons
Wojciech Lis <wlis@fb.com>
parents:
35431
diff
changeset
|
413 finishedthreads = [_t for _t in threads if not _t.is_alive()] |
86b8cc1f244e
worker: make windows workers daemons
Wojciech Lis <wlis@fb.com>
parents:
35431
diff
changeset
|
414 for t in finishedthreads: |
86b8cc1f244e
worker: make windows workers daemons
Wojciech Lis <wlis@fb.com>
parents:
35431
diff
changeset
|
415 if t.exception is not None: |
86b8cc1f244e
worker: make windows workers daemons
Wojciech Lis <wlis@fb.com>
parents:
35431
diff
changeset
|
416 raise t.exception |
86b8cc1f244e
worker: make windows workers daemons
Wojciech Lis <wlis@fb.com>
parents:
35431
diff
changeset
|
417 threads.remove(t) |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
418 except (Exception, KeyboardInterrupt): # re-raises |
35432
86b8cc1f244e
worker: make windows workers daemons
Wojciech Lis <wlis@fb.com>
parents:
35431
diff
changeset
|
419 trykillworkers() |
86b8cc1f244e
worker: make windows workers daemons
Wojciech Lis <wlis@fb.com>
parents:
35431
diff
changeset
|
420 raise |
35427
02b36e860e0b
workers: implemented worker on windows
Wojciech Lis <wlis@fb.com>
parents:
34646
diff
changeset
|
421 while not resultqueue.empty(): |
42522
d29db0a0c4eb
update: fix spurious unclean status bug shown by previous commit
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents:
42455
diff
changeset
|
422 res = resultqueue.get() |
42455
5ca136bbd3f6
worker: support parallelization of functions with return values
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
40989
diff
changeset
|
423 if hasretval and res[0]: |
42522
d29db0a0c4eb
update: fix spurious unclean status bug shown by previous commit
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents:
42455
diff
changeset
|
424 retval.update(res[1]) |
42455
5ca136bbd3f6
worker: support parallelization of functions with return values
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
40989
diff
changeset
|
425 else: |
5ca136bbd3f6
worker: support parallelization of functions with return values
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
40989
diff
changeset
|
426 yield res |
5ca136bbd3f6
worker: support parallelization of functions with return values
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents:
40989
diff
changeset
|
427 if hasretval: |
42522
d29db0a0c4eb
update: fix spurious unclean status bug shown by previous commit
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents:
42455
diff
changeset
|
428 yield True, retval |
35427
02b36e860e0b
workers: implemented worker on windows
Wojciech Lis <wlis@fb.com>
parents:
34646
diff
changeset
|
429 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
430 |
35427
02b36e860e0b
workers: implemented worker on windows
Wojciech Lis <wlis@fb.com>
parents:
34646
diff
changeset
|
431 if pycompat.iswindows: |
02b36e860e0b
workers: implemented worker on windows
Wojciech Lis <wlis@fb.com>
parents:
34646
diff
changeset
|
432 _platformworker = _windowsworker |
02b36e860e0b
workers: implemented worker on windows
Wojciech Lis <wlis@fb.com>
parents:
34646
diff
changeset
|
433 else: |
18638
047110c0e2a8
worker: allow a function to be run in multiple worker processes
Bryan O'Sullivan <bryano@fb.com>
parents:
18637
diff
changeset
|
434 _platformworker = _posixworker |
18707
d1a2b086d058
worker: on error, exit similarly to the first failing worker
Bryan O'Sullivan <bryano@fb.com>
parents:
18638
diff
changeset
|
435 _exitstatus = _posixexitstatus |
18638
047110c0e2a8
worker: allow a function to be run in multiple worker processes
Bryan O'Sullivan <bryano@fb.com>
parents:
18637
diff
changeset
|
436 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
42522
diff
changeset
|
437 |
18637
ac4dbceeb14a
worker: partition a list (of tasks) into equal-sized chunks
Bryan O'Sullivan <bryano@fb.com>
parents:
18636
diff
changeset
|
438 def partition(lst, nslices): |
45942
89a2afe31e82
formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents:
45825
diff
changeset
|
439 """partition a list into N slices of roughly equal size |
28181
f8efc8a3a991
worker: change partition strategy to every Nth element
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26587
diff
changeset
|
440 |
f8efc8a3a991
worker: change partition strategy to every Nth element
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26587
diff
changeset
|
441 The current strategy takes every Nth element from the input. If |
f8efc8a3a991
worker: change partition strategy to every Nth element
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26587
diff
changeset
|
442 we ever write workers that need to preserve grouping in input |
f8efc8a3a991
worker: change partition strategy to every Nth element
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26587
diff
changeset
|
443 we should consider allowing callers to specify a partition strategy. |
28292
3eb7faf6d958
worker: document poor partitioning scheme impact
Gregory Szorc <gregory.szorc@gmail.com>
parents:
28181
diff
changeset
|
444 |
46819
d4ba4d51f85f
contributor: change mentions of mpm to olivia
Raphaël Gomès <rgomes@octobus.net>
parents:
46698
diff
changeset
|
445 olivia is not a fan of this partitioning strategy when files are involved. |
28292
3eb7faf6d958
worker: document poor partitioning scheme impact
Gregory Szorc <gregory.szorc@gmail.com>
parents:
28181
diff
changeset
|
446 In his words: |
3eb7faf6d958
worker: document poor partitioning scheme impact
Gregory Szorc <gregory.szorc@gmail.com>
parents:
28181
diff
changeset
|
447 |
3eb7faf6d958
worker: document poor partitioning scheme impact
Gregory Szorc <gregory.szorc@gmail.com>
parents:
28181
diff
changeset
|
448 Single-threaded Mercurial makes a point of creating and visiting |
3eb7faf6d958
worker: document poor partitioning scheme impact
Gregory Szorc <gregory.szorc@gmail.com>
parents:
28181
diff
changeset
|
449 files in a fixed order (alphabetical). When creating files in order, |
3eb7faf6d958
worker: document poor partitioning scheme impact
Gregory Szorc <gregory.szorc@gmail.com>
parents:
28181
diff
changeset
|
450 a typical filesystem is likely to allocate them on nearby regions on |
3eb7faf6d958
worker: document poor partitioning scheme impact
Gregory Szorc <gregory.szorc@gmail.com>
parents:
28181
diff
changeset
|
451 disk. Thus, when revisiting in the same order, locality is maximized |
3eb7faf6d958
worker: document poor partitioning scheme impact
Gregory Szorc <gregory.szorc@gmail.com>
parents:
28181
diff
changeset
|
452 and various forms of OS and disk-level caching and read-ahead get a |
3eb7faf6d958
worker: document poor partitioning scheme impact
Gregory Szorc <gregory.szorc@gmail.com>
parents:
28181
diff
changeset
|
453 chance to work. |
3eb7faf6d958
worker: document poor partitioning scheme impact
Gregory Szorc <gregory.szorc@gmail.com>
parents:
28181
diff
changeset
|
454 |
3eb7faf6d958
worker: document poor partitioning scheme impact
Gregory Szorc <gregory.szorc@gmail.com>
parents:
28181
diff
changeset
|
455 This effect can be quite significant on spinning disks. I discovered it |
3eb7faf6d958
worker: document poor partitioning scheme impact
Gregory Szorc <gregory.szorc@gmail.com>
parents:
28181
diff
changeset
|
456 circa Mercurial v0.4 when revlogs were named by hashes of filenames. |
3eb7faf6d958
worker: document poor partitioning scheme impact
Gregory Szorc <gregory.szorc@gmail.com>
parents:
28181
diff
changeset
|
457 Tarring a repo and copying it to another disk effectively randomized |
3eb7faf6d958
worker: document poor partitioning scheme impact
Gregory Szorc <gregory.szorc@gmail.com>
parents:
28181
diff
changeset
|
458 the revlog ordering on disk by sorting the revlogs by hash and suddenly |
3eb7faf6d958
worker: document poor partitioning scheme impact
Gregory Szorc <gregory.szorc@gmail.com>
parents:
28181
diff
changeset
|
459 performance of my kernel checkout benchmark dropped by ~10x because the |
3eb7faf6d958
worker: document poor partitioning scheme impact
Gregory Szorc <gregory.szorc@gmail.com>
parents:
28181
diff
changeset
|
460 "working set" of sectors visited no longer fit in the drive's cache and |
3eb7faf6d958
worker: document poor partitioning scheme impact
Gregory Szorc <gregory.szorc@gmail.com>
parents:
28181
diff
changeset
|
461 the workload switched from streaming to random I/O. |
3eb7faf6d958
worker: document poor partitioning scheme impact
Gregory Szorc <gregory.szorc@gmail.com>
parents:
28181
diff
changeset
|
462 |
3eb7faf6d958
worker: document poor partitioning scheme impact
Gregory Szorc <gregory.szorc@gmail.com>
parents:
28181
diff
changeset
|
463 What we should really be doing is have workers read filenames from a |
3eb7faf6d958
worker: document poor partitioning scheme impact
Gregory Szorc <gregory.szorc@gmail.com>
parents:
28181
diff
changeset
|
464 ordered queue. This preserves locality and also keeps any worker from |
3eb7faf6d958
worker: document poor partitioning scheme impact
Gregory Szorc <gregory.szorc@gmail.com>
parents:
28181
diff
changeset
|
465 getting more than one file out of balance. |
45942
89a2afe31e82
formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents:
45825
diff
changeset
|
466 """ |
28181
f8efc8a3a991
worker: change partition strategy to every Nth element
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26587
diff
changeset
|
467 for i in range(nslices): |
f8efc8a3a991
worker: change partition strategy to every Nth element
Gregory Szorc <gregory.szorc@gmail.com>
parents:
26587
diff
changeset
|
468 yield lst[i::nslices] |