Mercurial > hg

changeset 39857:8dab7c8a93eb

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
upgrade: report size of backing files, not internal storage size upgrade.py is the only consumer of filelog.index, which I'd like to eliminate from the file storage interface. This commit changes the upgrade code to report the storage size of files by looking at the size of the files backing its storage instead of looking at the index. I'm not convinced the approach in this patch will live very long because it is relying on low-level attributes like "opener" and "files," which may behave very differently on non-revlog storage. But the data is only used for reporting purposes and it does get us one step closer to eliminating "index." A side-effect of this change is we now report the size of the revlog index data - not just the revision data. I think this is more accurate. Differential Revision: https://phab.mercurial-scm.org/D4717
author Gregory Szorc <gregory.szorc@gmail.com>
date Mon, 24 Sep 2018 09:37:19 -0700
parents 96838b620b9c
children 9534fe1e5d28
files mercurial/upgrade.py tests/test-lfs-serve.t tests/test-upgrade-repo.t
diffstat 3 files changed, 34 insertions(+), 32 deletions(-) [+]
line wrap: on
line diff
--- a/mercurial/upgrade.py	Thu Sep 20 18:07:42 2018 -0700
+++ b/mercurial/upgrade.py	Mon Sep 24 09:37:19 2018 -0700
@@ -487,10 +487,13 @@
 
         datasize = 0
         rawsize = 0
+
+        for path in rl.files():
+            datasize += rl.opener.stat(path).st_size
+
         idx = rl.index
         for rev in rl:
             e = idx[rev]
-            datasize += e[1]
             rawsize += e[2]
 
         srcsize += datasize
@@ -582,9 +585,8 @@
                     deltabothparents=deltabothparents)
 
         datasize = 0
-        idx = newrl.index
-        for rev in newrl:
-            datasize += idx[rev][1]
+        for path in newrl.files():
+            datasize += newrl.opener.stat(path).st_size
 
         dstsize += datasize
 
--- a/tests/test-lfs-serve.t	Thu Sep 20 18:07:42 2018 -0700
+++ b/tests/test-lfs-serve.t	Mon Sep 24 09:37:19 2018 -0700
@@ -114,12 +114,12 @@
   creating temporary repository to stage migrated data: * (glob)
   (it is safe to interrupt this process any time before data migration completes)
   migrating 3 total revisions (1 in filelogs, 1 in manifests, 1 in changelog)
-  migrating 132 bytes in store; 129 bytes tracked data
-  migrating 1 filelogs containing 1 revisions (9 bytes in store; 8 bytes tracked data)
+  migrating 324 bytes in store; 129 bytes tracked data
+  migrating 1 filelogs containing 1 revisions (73 bytes in store; 8 bytes tracked data)
   finished migrating 1 filelog revisions across 1 filelogs; change in size: 0 bytes
-  migrating 1 manifests containing 1 revisions (53 bytes in store; 52 bytes tracked data)
+  migrating 1 manifests containing 1 revisions (117 bytes in store; 52 bytes tracked data)
   finished migrating 1 manifest revisions across 1 manifests; change in size: 0 bytes
-  migrating changelog containing 1 revisions (70 bytes in store; 69 bytes tracked data)
+  migrating changelog containing 1 revisions (134 bytes in store; 69 bytes tracked data)
   finished migrating 1 changelog revisions; change in size: 0 bytes
   finished migrating 3 total revisions; total change in store size: 0 bytes
   copying phaseroots
--- a/tests/test-upgrade-repo.t	Thu Sep 20 18:07:42 2018 -0700
+++ b/tests/test-upgrade-repo.t	Mon Sep 24 09:37:19 2018 -0700
@@ -345,12 +345,12 @@
   creating temporary repository to stage migrated data: $TESTTMP/upgradegd/.hg/upgrade.* (glob)
   (it is safe to interrupt this process any time before data migration completes)
   migrating 9 total revisions (3 in filelogs, 3 in manifests, 3 in changelog)
-  migrating 341 bytes in store; 401 bytes tracked data
-  migrating 3 filelogs containing 3 revisions (0 bytes in store; 0 bytes tracked data)
+  migrating 917 bytes in store; 401 bytes tracked data
+  migrating 3 filelogs containing 3 revisions (192 bytes in store; 0 bytes tracked data)
   finished migrating 3 filelog revisions across 3 filelogs; change in size: 0 bytes
-  migrating 1 manifests containing 3 revisions (157 bytes in store; 220 bytes tracked data)
+  migrating 1 manifests containing 3 revisions (349 bytes in store; 220 bytes tracked data)
   finished migrating 3 manifest revisions across 1 manifests; change in size: 0 bytes
-  migrating changelog containing 3 revisions (184 bytes in store; 181 bytes tracked data)
+  migrating changelog containing 3 revisions (376 bytes in store; 181 bytes tracked data)
   finished migrating 3 changelog revisions; change in size: 0 bytes
   finished migrating 9 total revisions; total change in store size: 0 bytes
   copying phaseroots
@@ -442,12 +442,12 @@
   creating temporary repository to stage migrated data: $TESTTMP/store-filenames/.hg/upgrade.* (glob)
   (it is safe to interrupt this process any time before data migration completes)
   migrating 3 total revisions (1 in filelogs, 1 in manifests, 1 in changelog)
-  migrating 109 bytes in store; 107 bytes tracked data
-  migrating 1 filelogs containing 1 revisions (0 bytes in store; 0 bytes tracked data)
+  migrating 301 bytes in store; 107 bytes tracked data
+  migrating 1 filelogs containing 1 revisions (64 bytes in store; 0 bytes tracked data)
   finished migrating 1 filelog revisions across 1 filelogs; change in size: 0 bytes
-  migrating 1 manifests containing 1 revisions (46 bytes in store; 45 bytes tracked data)
+  migrating 1 manifests containing 1 revisions (110 bytes in store; 45 bytes tracked data)
   finished migrating 1 manifest revisions across 1 manifests; change in size: 0 bytes
-  migrating changelog containing 1 revisions (63 bytes in store; 62 bytes tracked data)
+  migrating changelog containing 1 revisions (127 bytes in store; 62 bytes tracked data)
   finished migrating 1 changelog revisions; change in size: 0 bytes
   finished migrating 3 total revisions; total change in store size: 0 bytes
   copying .XX_special_filename
@@ -476,12 +476,12 @@
   creating temporary repository to stage migrated data: $TESTTMP/store-filenames/.hg/upgrade.* (glob)
   (it is safe to interrupt this process any time before data migration completes)
   migrating 3 total revisions (1 in filelogs, 1 in manifests, 1 in changelog)
-  migrating 109 bytes in store; 107 bytes tracked data
-  migrating 1 filelogs containing 1 revisions (0 bytes in store; 0 bytes tracked data)
+  migrating 301 bytes in store; 107 bytes tracked data
+  migrating 1 filelogs containing 1 revisions (64 bytes in store; 0 bytes tracked data)
   finished migrating 1 filelog revisions across 1 filelogs; change in size: 0 bytes
-  migrating 1 manifests containing 1 revisions (46 bytes in store; 45 bytes tracked data)
+  migrating 1 manifests containing 1 revisions (110 bytes in store; 45 bytes tracked data)
   finished migrating 1 manifest revisions across 1 manifests; change in size: 0 bytes
-  migrating changelog containing 1 revisions (63 bytes in store; 62 bytes tracked data)
+  migrating changelog containing 1 revisions (127 bytes in store; 62 bytes tracked data)
   finished migrating 1 changelog revisions; change in size: 0 bytes
   finished migrating 3 total revisions; total change in store size: 0 bytes
   copying .XX_special_filename
@@ -531,12 +531,12 @@
   creating temporary repository to stage migrated data: $TESTTMP/largefilesrepo/.hg/upgrade.* (glob)
   (it is safe to interrupt this process any time before data migration completes)
   migrating 3 total revisions (1 in filelogs, 1 in manifests, 1 in changelog)
-  migrating 163 bytes in store; 160 bytes tracked data
-  migrating 1 filelogs containing 1 revisions (42 bytes in store; 41 bytes tracked data)
+  migrating 355 bytes in store; 160 bytes tracked data
+  migrating 1 filelogs containing 1 revisions (106 bytes in store; 41 bytes tracked data)
   finished migrating 1 filelog revisions across 1 filelogs; change in size: 0 bytes
-  migrating 1 manifests containing 1 revisions (52 bytes in store; 51 bytes tracked data)
+  migrating 1 manifests containing 1 revisions (116 bytes in store; 51 bytes tracked data)
   finished migrating 1 manifest revisions across 1 manifests; change in size: 0 bytes
-  migrating changelog containing 1 revisions (69 bytes in store; 68 bytes tracked data)
+  migrating changelog containing 1 revisions (133 bytes in store; 68 bytes tracked data)
   finished migrating 1 changelog revisions; change in size: 0 bytes
   finished migrating 3 total revisions; total change in store size: 0 bytes
   copying phaseroots
@@ -583,12 +583,12 @@
   creating temporary repository to stage migrated data: $TESTTMP/largefilesrepo/.hg/upgrade.* (glob)
   (it is safe to interrupt this process any time before data migration completes)
   migrating 6 total revisions (2 in filelogs, 2 in manifests, 2 in changelog)
-  migrating 417 bytes in store; 467 bytes tracked data
-  migrating 2 filelogs containing 2 revisions (168 bytes in store; 182 bytes tracked data)
+  migrating 801 bytes in store; 467 bytes tracked data
+  migrating 2 filelogs containing 2 revisions (296 bytes in store; 182 bytes tracked data)
   finished migrating 2 filelog revisions across 2 filelogs; change in size: 0 bytes
-  migrating 1 manifests containing 2 revisions (113 bytes in store; 151 bytes tracked data)
+  migrating 1 manifests containing 2 revisions (241 bytes in store; 151 bytes tracked data)
   finished migrating 2 manifest revisions across 1 manifests; change in size: 0 bytes
-  migrating changelog containing 2 revisions (136 bytes in store; 134 bytes tracked data)
+  migrating changelog containing 2 revisions (264 bytes in store; 134 bytes tracked data)
   finished migrating 2 changelog revisions; change in size: 0 bytes
   finished migrating 6 total revisions; total change in store size: 0 bytes
   copying phaseroots
@@ -681,12 +681,12 @@
   creating temporary repository to stage migrated data: $TESTTMP/localconfig/.hg/upgrade.* (glob)
   (it is safe to interrupt this process any time before data migration completes)
   migrating 9 total revisions (3 in filelogs, 3 in manifests, 3 in changelog)
-  migrating 497 bytes in store; 882 bytes tracked data
-  migrating 1 filelogs containing 3 revisions (182 bytes in store; 573 bytes tracked data)
+  migrating 1.05 KB in store; 882 bytes tracked data
+  migrating 1 filelogs containing 3 revisions (374 bytes in store; 573 bytes tracked data)
   finished migrating 3 filelog revisions across 1 filelogs; change in size: -63 bytes
-  migrating 1 manifests containing 3 revisions (141 bytes in store; 138 bytes tracked data)
+  migrating 1 manifests containing 3 revisions (333 bytes in store; 138 bytes tracked data)
   finished migrating 3 manifest revisions across 1 manifests; change in size: 0 bytes
-  migrating changelog containing 3 revisions (174 bytes in store; 171 bytes tracked data)
+  migrating changelog containing 3 revisions (366 bytes in store; 171 bytes tracked data)
   finished migrating 3 changelog revisions; change in size: 0 bytes
   finished migrating 9 total revisions; total change in store size: -63 bytes
   copying phaseroots