annotate hgext/largefiles/localstore.py @ 36438:93228b2a1fc0

run-tests: don't mask errors when a server fails to start There are sporadic instances of this on Windows. They seem to happen more frequently after the test machine is rebooted, although the only way to hit it on my laptop is to loop certain tests with -j9 for hours. The problem with masking out the specific failure is that there's no way to know if it's the same line in the test that's failing, or if it is random. The justification for adding this masking in 52e9e63f1495 was that the failures occur regularly, but that's not the case anymore. The port number is still printed, in case that turns out to be useful.
author Matt Harbison <matt_harbison@yahoo.com>
date Sun, 25 Feb 2018 21:04:12 -0500
parents 736f92c44656
children 2372284d9457
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
1 # Copyright 2009-2010 Gregory P. Ward
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
2 # Copyright 2009-2010 Intelerad Medical Systems Incorporated
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
3 # Copyright 2010-2011 Fog Creek Software
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
4 # Copyright 2010-2011 Unity Technologies
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
5 #
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
6 # This software may be used and distributed according to the terms of the
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
7 # GNU General Public License version 2 or any later version.
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
8
15252
6e809bb4f969 largefiles: improve comments, internal docstrings
Greg Ward <greg@gerg.ca>
parents: 15168
diff changeset
9 '''store class for local filesystem'''
29310
f89f83c8393a py3: make largefiles/localstore.py use absolute_import
liscju <piotr.listkiewicz@gmail.com>
parents: 29067
diff changeset
10 from __future__ import absolute_import
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
11
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
12 from mercurial.i18n import _
30180
736f92c44656 largefiles: always use filechunkiter when iterating files
Mads Kiilerich <madski@unity3d.com>
parents: 29421
diff changeset
13 from mercurial import util
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
14
29310
f89f83c8393a py3: make largefiles/localstore.py use absolute_import
liscju <piotr.listkiewicz@gmail.com>
parents: 29067
diff changeset
15 from . import (
f89f83c8393a py3: make largefiles/localstore.py use absolute_import
liscju <piotr.listkiewicz@gmail.com>
parents: 29067
diff changeset
16 basestore,
f89f83c8393a py3: make largefiles/localstore.py use absolute_import
liscju <piotr.listkiewicz@gmail.com>
parents: 29067
diff changeset
17 lfutil,
f89f83c8393a py3: make largefiles/localstore.py use absolute_import
liscju <piotr.listkiewicz@gmail.com>
parents: 29067
diff changeset
18 )
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
19
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
20 class localstore(basestore.basestore):
15317
41f371150ccb largefiles: make the store primary, and the user cache secondary
Benjamin Pollack <benjamin@bitquabit.com>
parents: 15316
diff changeset
21 '''localstore first attempts to grab files out of the store in the remote
17424
e7cfe3587ea4 fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents: 17191
diff changeset
22 Mercurial repository. Failing that, it attempts to grab the files from
15317
41f371150ccb largefiles: make the store primary, and the user cache secondary
Benjamin Pollack <benjamin@bitquabit.com>
parents: 15316
diff changeset
23 the user cache.'''
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
24
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
25 def __init__(self, ui, repo, remote):
17191
5884812686f7 peer: introduce peer methods to prepare for peer classes
Sune Foldager <cryo@cyanite.org>
parents: 16928
diff changeset
26 self.remote = remote.local()
18155
5206af8894a3 largefiles: cleanup of warnings on errors getting largefiles
Mads Kiilerich <madski@unity3d.com>
parents: 17439
diff changeset
27 super(localstore, self).__init__(ui, repo, self.remote.url())
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
28
15317
41f371150ccb largefiles: make the store primary, and the user cache secondary
Benjamin Pollack <benjamin@bitquabit.com>
parents: 15316
diff changeset
29 def put(self, source, hash):
41f371150ccb largefiles: make the store primary, and the user cache secondary
Benjamin Pollack <benjamin@bitquabit.com>
parents: 15316
diff changeset
30 if lfutil.instore(self.remote, hash):
41f371150ccb largefiles: make the store primary, and the user cache secondary
Benjamin Pollack <benjamin@bitquabit.com>
parents: 15316
diff changeset
31 return
19007
266b5fb72f26 largefiles: 'put' should store 'source' file in under 'hash', also in localstore
Mads Kiilerich <madski@unity3d.com>
parents: 19003
diff changeset
32 lfutil.link(source, lfutil.storepath(self.remote, hash))
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
33
17411
a02e36568e88 largefiles: adjust localstore to handle batch statlfile requests (issue3583)
Matt Harbison <matt_harbison@yahoo.com>
parents: 17191
diff changeset
34 def exists(self, hashes):
a02e36568e88 largefiles: adjust localstore to handle batch statlfile requests (issue3583)
Matt Harbison <matt_harbison@yahoo.com>
parents: 17191
diff changeset
35 retval = {}
a02e36568e88 largefiles: adjust localstore to handle batch statlfile requests (issue3583)
Matt Harbison <matt_harbison@yahoo.com>
parents: 17191
diff changeset
36 for hash in hashes:
a02e36568e88 largefiles: adjust localstore to handle batch statlfile requests (issue3583)
Matt Harbison <matt_harbison@yahoo.com>
parents: 17191
diff changeset
37 retval[hash] = lfutil.instore(self.remote, hash)
a02e36568e88 largefiles: adjust localstore to handle batch statlfile requests (issue3583)
Matt Harbison <matt_harbison@yahoo.com>
parents: 17191
diff changeset
38 return retval
a02e36568e88 largefiles: adjust localstore to handle batch statlfile requests (issue3583)
Matt Harbison <matt_harbison@yahoo.com>
parents: 17191
diff changeset
39
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
40 def _getfile(self, tmpfile, filename, hash):
19000
eaf146e811a4 largefiles: refactoring - use findfile in localstore._getfile
Mads Kiilerich <madski@unity3d.com>
parents: 18998
diff changeset
41 path = lfutil.findfile(self.remote, hash)
eaf146e811a4 largefiles: refactoring - use findfile in localstore._getfile
Mads Kiilerich <madski@unity3d.com>
parents: 18998
diff changeset
42 if not path:
18155
5206af8894a3 largefiles: cleanup of warnings on errors getting largefiles
Mads Kiilerich <madski@unity3d.com>
parents: 17439
diff changeset
43 raise basestore.StoreError(filename, hash, self.url,
16928
73b9286e667c largefiles: lowercase messages
Martin Geisler <mg@aragost.com>
parents: 15371
diff changeset
44 _("can't get file locally"))
27769
40bd01be5c25 largefiles: use a context manager in _getfile
Bryan O'Sullivan <bryano@fb.com>
parents: 24630
diff changeset
45 with open(path, 'rb') as fd:
30180
736f92c44656 largefiles: always use filechunkiter when iterating files
Mads Kiilerich <madski@unity3d.com>
parents: 29421
diff changeset
46 return lfutil.copyandhash(
736f92c44656 largefiles: always use filechunkiter when iterating files
Mads Kiilerich <madski@unity3d.com>
parents: 29421
diff changeset
47 util.filechunkiter(fd), tmpfile)
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
48
29067
207c0db08953 largefiles: change basestore._verifyfile to take list of files to check
liscju <piotr.listkiewicz@gmail.com>
parents: 27769
diff changeset
49 def _verifyfiles(self, contents, filestocheck):
207c0db08953 largefiles: change basestore._verifyfile to take list of files to check
liscju <piotr.listkiewicz@gmail.com>
parents: 27769
diff changeset
50 failed = False
207c0db08953 largefiles: change basestore._verifyfile to take list of files to check
liscju <piotr.listkiewicz@gmail.com>
parents: 27769
diff changeset
51 for cset, filename, expectedhash in filestocheck:
29421
ecbbf4d56ee8 largefiles: check file in the repo store before checking remotely (issue5257)
liscju <piotr.listkiewicz@gmail.com>
parents: 29409
diff changeset
52 storepath, exists = lfutil.findstorepath(self.repo, expectedhash)
ecbbf4d56ee8 largefiles: check file in the repo store before checking remotely (issue5257)
liscju <piotr.listkiewicz@gmail.com>
parents: 29409
diff changeset
53 if not exists:
ecbbf4d56ee8 largefiles: check file in the repo store before checking remotely (issue5257)
liscju <piotr.listkiewicz@gmail.com>
parents: 29409
diff changeset
54 storepath, exists = lfutil.findstorepath(
ecbbf4d56ee8 largefiles: check file in the repo store before checking remotely (issue5257)
liscju <piotr.listkiewicz@gmail.com>
parents: 29409
diff changeset
55 self.remote, expectedhash)
29067
207c0db08953 largefiles: change basestore._verifyfile to take list of files to check
liscju <piotr.listkiewicz@gmail.com>
parents: 27769
diff changeset
56 if not exists:
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
57 self.ui.warn(
29067
207c0db08953 largefiles: change basestore._verifyfile to take list of files to check
liscju <piotr.listkiewicz@gmail.com>
parents: 27769
diff changeset
58 _('changeset %s: %s references missing %s\n')
18545
a49b7c9fc246 largefiles: report localstore errors with single line warnings messages
Mads Kiilerich <madski@unity3d.com>
parents: 18155
diff changeset
59 % (cset, filename, storepath))
29067
207c0db08953 largefiles: change basestore._verifyfile to take list of files to check
liscju <piotr.listkiewicz@gmail.com>
parents: 27769
diff changeset
60 failed = True
207c0db08953 largefiles: change basestore._verifyfile to take list of files to check
liscju <piotr.listkiewicz@gmail.com>
parents: 27769
diff changeset
61 elif contents:
207c0db08953 largefiles: change basestore._verifyfile to take list of files to check
liscju <piotr.listkiewicz@gmail.com>
parents: 27769
diff changeset
62 actualhash = lfutil.hashfile(storepath)
207c0db08953 largefiles: change basestore._verifyfile to take list of files to check
liscju <piotr.listkiewicz@gmail.com>
parents: 27769
diff changeset
63 if actualhash != expectedhash:
207c0db08953 largefiles: change basestore._verifyfile to take list of files to check
liscju <piotr.listkiewicz@gmail.com>
parents: 27769
diff changeset
64 self.ui.warn(
207c0db08953 largefiles: change basestore._verifyfile to take list of files to check
liscju <piotr.listkiewicz@gmail.com>
parents: 27769
diff changeset
65 _('changeset %s: %s references corrupted %s\n')
207c0db08953 largefiles: change basestore._verifyfile to take list of files to check
liscju <piotr.listkiewicz@gmail.com>
parents: 27769
diff changeset
66 % (cset, filename, storepath))
207c0db08953 largefiles: change basestore._verifyfile to take list of files to check
liscju <piotr.listkiewicz@gmail.com>
parents: 27769
diff changeset
67 failed = True
207c0db08953 largefiles: change basestore._verifyfile to take list of files to check
liscju <piotr.listkiewicz@gmail.com>
parents: 27769
diff changeset
68 return failed