comparison rust/hg-core/src/dirstate/status.rs @ 48068:bf8837e3d7ce

dirstate: Remove the flat Rust DirstateMap implementation Before this changeset we had two Rust implementations of `DirstateMap`. This removes the "flat" DirstateMap so that the "tree" DirstateMap is always used when Rust enabled. This simplifies the code a lot, and will enable (in the next changeset) further removal of a trait abstraction. This is a performance regression when: * Rust is enabled, and * The repository uses the legacy dirstate-v1 file format, and * For `hg status`, unknown files are not listed (such as with `-mard`) The regression is about 100 milliseconds for `hg status -mard` on a semi-large repository (mozilla-central), from ~320ms to ~420ms. We deem this to be small enough to be worth it. The new dirstate-v2 is still experimental at this point, but we aim to stabilize it (though not yet enable it by default for new repositories) in Mercurial 6.0. Eventually, upgrating repositories to dirsate-v2 will eliminate this regression (and enable other performance improvements). # Background The flat DirstateMap was introduced with the first Rust implementation of the status algorithm. It works similarly to the previous Python + C one, with a single `HashMap` that associates file paths to a `DirstateEntry` (where Python has a dict). We later added the tree DirstateMap where the root of the tree contains nodes for files and directories that are directly at the root of the repository, and nodes for directories can contain child nodes representing the files and directly that *they* contain directly. The shape of this tree mirrors that of the working directory in the filesystem. This enables the status algorithm to traverse this tree in tandem with traversing the filesystem tree, which in turns enables a more efficient algorithm. Furthermore, the new dirstate-v2 file format is also based on a tree of the same shape. The tree DirstateMap can access a dirstate-v2 file without parsing it: binary data in a single large (possibly memory-mapped) bytes buffer is traversed on demand. This allows `DirstateMap` creation to take `O(1)` time. (Mutation works by creating new in-memory nodes with copy-on-write semantics, and serialization is append-mostly.) The tradeoff is that for "legacy" repositories that use the dirstate-v1 file format, parsing that file into a tree DirstateMap takes more time. Profiling shows that this time is dominated by `HashMap`. For a dirstate containing `F` files with an average `D` directory depth, the flat DirstateMap does parsing in `O(F)` number of HashMap operations but the tree DirstateMap in `O(F × D)` operations, since each node has its own HashMap containing its child nodes. This slower costs ~140ms on an old snapshot of mozilla-central, and ~80ms on an old snapshot of the Netbeans repository. The status algorithm is faster, but with `-mard` (when not listing unknown files) it is typically not faster *enough* to compensate the slower parsing. Both Rust implementations are always faster than the Python + C implementation # Benchmark results All benchmarks are run on changeset 98c0408324e6, with repositories that use the dirstate-v1 file format, on a server with 4 CPU cores and 4 CPU threads (no HyperThreading). `hg status` benchmarks show wall clock times of the entire command as the average and standard deviation of serveral runs, collected by https://github.com/sharkdp/hyperfine and reformated. Parsing benchmarks are wall clock time of the Rust function that converts a bytes buffer of the dirstate file into the `DirstateMap` data structure as used by the status algorithm. A single run each, collected by running `hg status` this environment variable: RUST_LOG=hg::dirstate::dirstate_map=trace,hg::dirstate_tree::dirstate_map=trace Benchmark 1: Rust flat DirstateMap → Rust tree DirstateMap hg status mozilla-clean 562.3 ms ± 2.0 ms → 462.5 ms ± 0.6 ms 1.22 ± 0.00 times faster mozilla-dirty 859.6 ms ± 2.2 ms → 719.5 ms ± 3.2 ms 1.19 ± 0.01 times faster mozilla-ignored 558.2 ms ± 3.0 ms → 457.9 ms ± 2.9 ms 1.22 ± 0.01 times faster mozilla-unknowns 859.4 ms ± 5.7 ms → 716.0 ms ± 4.7 ms 1.20 ± 0.01 times faster netbeans-clean 336.5 ms ± 0.9 ms → 339.5 ms ± 0.4 ms 0.99 ± 0.00 times faster netbeans-dirty 491.4 ms ± 1.6 ms → 475.1 ms ± 1.2 ms 1.03 ± 0.00 times faster netbeans-ignored 343.7 ms ± 1.0 ms → 347.8 ms ± 0.4 ms 0.99 ± 0.00 times faster netbeans-unknowns 484.3 ms ± 1.0 ms → 466.0 ms ± 1.2 ms 1.04 ± 0.00 times faster hg status -mard mozilla-clean 317.3 ms ± 0.6 ms → 422.5 ms ± 1.2 ms 0.75 ± 0.00 times faster mozilla-dirty 315.4 ms ± 0.6 ms → 417.7 ms ± 1.1 ms 0.76 ± 0.00 times faster mozilla-ignored 314.6 ms ± 0.6 ms → 417.4 ms ± 1.0 ms 0.75 ± 0.00 times faster mozilla-unknowns 312.9 ms ± 0.9 ms → 417.3 ms ± 1.6 ms 0.75 ± 0.00 times faster netbeans-clean 212.0 ms ± 0.6 ms → 283.6 ms ± 0.8 ms 0.75 ± 0.00 times faster netbeans-dirty 211.4 ms ± 1.0 ms → 283.4 ms ± 1.6 ms 0.75 ± 0.01 times faster netbeans-ignored 211.4 ms ± 0.9 ms → 283.9 ms ± 0.8 ms 0.74 ± 0.01 times faster netbeans-unknowns 211.1 ms ± 0.6 ms → 283.4 ms ± 1.0 ms 0.74 ± 0.00 times faster Parsing mozilla-clean 38.4ms → 177.6ms mozilla-dirty 38.8ms → 177.0ms mozilla-ignored 38.8ms → 178.0ms mozilla-unknowns 38.7ms → 176.9ms netbeans-clean 16.5ms → 97.3ms netbeans-dirty 16.5ms → 98.4ms netbeans-ignored 16.9ms → 97.4ms netbeans-unknowns 16.9ms → 96.3ms Benchmark 2: Python + C dirstatemap → Rust tree DirstateMap hg status mozilla-clean 1261.0 ms ± 3.6 ms → 461.1 ms ± 0.5 ms 2.73 ± 0.00 times faster mozilla-dirty 2293.4 ms ± 9.1 ms → 719.6 ms ± 3.6 ms 3.19 ± 0.01 times faster mozilla-ignored 1240.4 ms ± 2.3 ms → 457.7 ms ± 1.9 ms 2.71 ± 0.00 times faster mozilla-unknowns 2283.3 ms ± 9.0 ms → 719.7 ms ± 3.8 ms 3.17 ± 0.01 times faster netbeans-clean 879.7 ms ± 3.5 ms → 339.9 ms ± 0.5 ms 2.59 ± 0.00 times faster netbeans-dirty 1257.3 ms ± 4.7 ms → 474.6 ms ± 1.6 ms 2.65 ± 0.01 times faster netbeans-ignored 943.9 ms ± 1.9 ms → 347.3 ms ± 1.1 ms 2.72 ± 0.00 times faster netbeans-unknowns 1188.1 ms ± 5.0 ms → 465.2 ms ± 2.3 ms 2.55 ± 0.01 times faster hg status -mard mozilla-clean 903.2 ms ± 3.6 ms → 423.4 ms ± 2.2 ms 2.13 ± 0.01 times faster mozilla-dirty 884.6 ms ± 4.5 ms → 417.3 ms ± 1.4 ms 2.12 ± 0.01 times faster mozilla-ignored 881.9 ms ± 1.3 ms → 417.3 ms ± 0.8 ms 2.11 ± 0.00 times faster mozilla-unknowns 878.5 ms ± 1.9 ms → 416.4 ms ± 0.9 ms 2.11 ± 0.00 times faster netbeans-clean 434.9 ms ± 1.8 ms → 284.0 ms ± 0.8 ms 1.53 ± 0.01 times faster netbeans-dirty 434.1 ms ± 0.8 ms → 283.1 ms ± 0.8 ms 1.53 ± 0.00 times faster netbeans-ignored 431.7 ms ± 1.1 ms → 283.6 ms ± 1.8 ms 1.52 ± 0.01 times faster netbeans-unknowns 433.0 ms ± 1.3 ms → 283.5 ms ± 0.7 ms 1.53 ± 0.00 times faster Differential Revision: https://phab.mercurial-scm.org/D11516
author Simon Sapin <simon.sapin@octobus.net>
date Mon, 27 Sep 2021 12:09:15 +0200
parents 1b2ee68e85f9
children 269ff8978086
comparison
equal deleted inserted replaced
48067:d3eb5f50052c 48068:bf8837e3d7ce
8 //! Rust implementation of dirstate.status (dirstate.py). 8 //! Rust implementation of dirstate.status (dirstate.py).
9 //! It is currently missing a lot of functionality compared to the Python one 9 //! It is currently missing a lot of functionality compared to the Python one
10 //! and will only be triggered in narrow cases. 10 //! and will only be triggered in narrow cases.
11 11
12 use crate::dirstate_tree::on_disk::DirstateV2ParseError; 12 use crate::dirstate_tree::on_disk::DirstateV2ParseError;
13 use crate::utils::path_auditor::PathAuditor; 13
14 use crate::{ 14 use crate::{
15 dirstate::SIZE_FROM_OTHER_PARENT, 15 utils::hg_path::{HgPath, HgPathError},
16 filepatterns::PatternFileWarning,
17 matchers::{get_ignore_function, Matcher, VisitChildrenSet},
18 utils::{
19 files::{find_dirs, HgMetadata},
20 hg_path::{
21 hg_path_to_path_buf, os_string_to_hg_path_buf, HgPath, HgPathBuf,
22 HgPathError,
23 },
24 },
25 CopyMap, DirstateEntry, DirstateMap, EntryState, FastHashMap,
26 PatternError, 16 PatternError,
27 }; 17 };
28 use lazy_static::lazy_static; 18
29 use micro_timer::timed; 19 use std::{borrow::Cow, fmt};
30 use rayon::prelude::*;
31 use std::{
32 borrow::Cow,
33 collections::HashSet,
34 fmt,
35 fs::{read_dir, DirEntry},
36 io::ErrorKind,
37 ops::Deref,
38 path::{Path, PathBuf},
39 };
40 20
41 /// Wrong type of file from a `BadMatch` 21 /// Wrong type of file from a `BadMatch`
42 /// Note: a lot of those don't exist on all platforms. 22 /// Note: a lot of those don't exist on all platforms.
43 #[derive(Debug, Copy, Clone)] 23 #[derive(Debug, Copy, Clone)]
44 pub enum BadType { 24 pub enum BadType {
68 pub enum BadMatch { 48 pub enum BadMatch {
69 OsError(i32), 49 OsError(i32),
70 BadType(BadType), 50 BadType(BadType),
71 } 51 }
72 52
73 /// Enum used to dispatch new status entries into the right collections.
74 /// Is similar to `crate::EntryState`, but represents the transient state of
75 /// entries during the lifetime of a command.
76 #[derive(Debug, Copy, Clone)]
77 pub enum Dispatch {
78 Unsure,
79 Modified,
80 Added,
81 Removed,
82 Deleted,
83 Clean,
84 Unknown,
85 Ignored,
86 /// Empty dispatch, the file is not worth listing
87 None,
88 /// Was explicitly matched but cannot be found/accessed
89 Bad(BadMatch),
90 Directory {
91 /// True if the directory used to be a file in the dmap so we can say
92 /// that it's been removed.
93 was_file: bool,
94 },
95 }
96
97 type IoResult<T> = std::io::Result<T>;
98
99 /// `Box<dyn Trait>` is syntactic sugar for `Box<dyn Trait + 'static>`, so add 53 /// `Box<dyn Trait>` is syntactic sugar for `Box<dyn Trait + 'static>`, so add
100 /// an explicit lifetime here to not fight `'static` bounds "out of nowhere". 54 /// an explicit lifetime here to not fight `'static` bounds "out of nowhere".
101 pub type IgnoreFnType<'a> = 55 pub type IgnoreFnType<'a> =
102 Box<dyn for<'r> Fn(&'r HgPath) -> bool + Sync + 'a>; 56 Box<dyn for<'r> Fn(&'r HgPath) -> bool + Sync + 'a>;
103 57
104 /// We have a good mix of owned (from directory traversal) and borrowed (from 58 /// We have a good mix of owned (from directory traversal) and borrowed (from
105 /// the dirstate/explicit) paths, this comes up a lot. 59 /// the dirstate/explicit) paths, this comes up a lot.
106 pub type HgPathCow<'a> = Cow<'a, HgPath>; 60 pub type HgPathCow<'a> = Cow<'a, HgPath>;
107
108 /// A path with its computed ``Dispatch`` information
109 type DispatchedPath<'a> = (HgPathCow<'a>, Dispatch);
110
111 /// The conversion from `HgPath` to a real fs path failed.
112 /// `22` is the error code for "Invalid argument"
113 const INVALID_PATH_DISPATCH: Dispatch = Dispatch::Bad(BadMatch::OsError(22));
114
115 /// Dates and times that are outside the 31-bit signed range are compared
116 /// modulo 2^31. This should prevent hg from behaving badly with very large
117 /// files or corrupt dates while still having a high probability of detecting
118 /// changes. (issue2608)
119 /// TODO I haven't found a way of having `b` be `Into<i32>`, since `From<u64>`
120 /// is not defined for `i32`, and there is no `As` trait. This forces the
121 /// caller to cast `b` as `i32`.
122 fn mod_compare(a: i32, b: i32) -> bool {
123 a & i32::max_value() != b & i32::max_value()
124 }
125
126 /// Return a sorted list containing information about the entries
127 /// in the directory.
128 ///
129 /// * `skip_dot_hg` - Return an empty vec if `path` contains a `.hg` directory
130 fn list_directory(
131 path: impl AsRef<Path>,
132 skip_dot_hg: bool,
133 ) -> std::io::Result<Vec<(HgPathBuf, DirEntry)>> {
134 let mut results = vec![];
135 let entries = read_dir(path.as_ref())?;
136
137 for entry in entries {
138 let entry = entry?;
139 let filename = os_string_to_hg_path_buf(entry.file_name())?;
140 let file_type = entry.file_type()?;
141 if skip_dot_hg && filename.as_bytes() == b".hg" && file_type.is_dir() {
142 return Ok(vec![]);
143 } else {
144 results.push((filename, entry))
145 }
146 }
147
148 results.sort_unstable_by_key(|e| e.0.clone());
149 Ok(results)
150 }
151
152 /// The file corresponding to the dirstate entry was found on the filesystem.
153 fn dispatch_found(
154 filename: impl AsRef<HgPath>,
155 entry: DirstateEntry,
156 metadata: HgMetadata,
157 copy_map: &CopyMap,
158 options: StatusOptions,
159 ) -> Dispatch {
160 match entry.state() {
161 EntryState::Normal => {
162 let mode = entry.mode();
163 let size = entry.size();
164 let mtime = entry.mtime();
165
166 let HgMetadata {
167 st_mode,
168 st_size,
169 st_mtime,
170 ..
171 } = metadata;
172
173 let size_changed = mod_compare(size, st_size as i32);
174 let mode_changed =
175 (mode ^ st_mode as i32) & 0o100 != 0o000 && options.check_exec;
176 let metadata_changed = size >= 0 && (size_changed || mode_changed);
177 let other_parent = size == SIZE_FROM_OTHER_PARENT;
178
179 if metadata_changed
180 || other_parent
181 || copy_map.contains_key(filename.as_ref())
182 {
183 if metadata.is_symlink() && size_changed {
184 // issue6456: Size returned may be longer due to encryption
185 // on EXT-4 fscrypt. TODO maybe only do it on EXT4?
186 Dispatch::Unsure
187 } else {
188 Dispatch::Modified
189 }
190 } else if mod_compare(mtime, st_mtime as i32)
191 || st_mtime == options.last_normal_time
192 {
193 // the file may have just been marked as normal and
194 // it may have changed in the same second without
195 // changing its size. This can happen if we quickly
196 // do multiple commits. Force lookup, so we don't
197 // miss such a racy file change.
198 Dispatch::Unsure
199 } else if options.list_clean {
200 Dispatch::Clean
201 } else {
202 Dispatch::None
203 }
204 }
205 EntryState::Merged => Dispatch::Modified,
206 EntryState::Added => Dispatch::Added,
207 EntryState::Removed => Dispatch::Removed,
208 }
209 }
210
211 /// The file corresponding to this Dirstate entry is missing.
212 fn dispatch_missing(state: EntryState) -> Dispatch {
213 match state {
214 // File was removed from the filesystem during commands
215 EntryState::Normal | EntryState::Merged | EntryState::Added => {
216 Dispatch::Deleted
217 }
218 // File was removed, everything is normal
219 EntryState::Removed => Dispatch::Removed,
220 }
221 }
222
223 fn dispatch_os_error(e: &std::io::Error) -> Dispatch {
224 Dispatch::Bad(BadMatch::OsError(
225 e.raw_os_error().expect("expected real OS error"),
226 ))
227 }
228
229 lazy_static! {
230 static ref DEFAULT_WORK: HashSet<&'static HgPath> = {
231 let mut h = HashSet::new();
232 h.insert(HgPath::new(b""));
233 h
234 };
235 }
236 61
237 #[derive(Debug, Copy, Clone)] 62 #[derive(Debug, Copy, Clone)]
238 pub struct StatusOptions { 63 pub struct StatusOptions {
239 /// Remember the most recent modification timeslot for status, to make 64 /// Remember the most recent modification timeslot for status, to make
240 /// sure we won't miss future size-preserving file content modifications 65 /// sure we won't miss future size-preserving file content modifications
317 f.write_str("dirstate-v2 parse error") 142 f.write_str("dirstate-v2 parse error")
318 } 143 }
319 } 144 }
320 } 145 }
321 } 146 }
322
323 /// Gives information about which files are changed in the working directory
324 /// and how, compared to the revision we're based on
325 pub struct Status<'a, M: ?Sized + Matcher + Sync> {
326 dmap: &'a DirstateMap,
327 pub(crate) matcher: &'a M,
328 root_dir: PathBuf,
329 pub(crate) options: StatusOptions,
330 ignore_fn: IgnoreFnType<'a>,
331 }
332
333 impl<'a, M> Status<'a, M>
334 where
335 M: ?Sized + Matcher + Sync,
336 {
337 pub fn new(
338 dmap: &'a DirstateMap,
339 matcher: &'a M,
340 root_dir: PathBuf,
341 ignore_files: Vec<PathBuf>,
342 options: StatusOptions,
343 ) -> StatusResult<(Self, Vec<PatternFileWarning>)> {
344 // Needs to outlive `dir_ignore_fn` since it's captured.
345
346 let (ignore_fn, warnings): (IgnoreFnType, _) =
347 if options.list_ignored || options.list_unknown {
348 get_ignore_function(ignore_files, &root_dir, &mut |_| {})?
349 } else {
350 (Box::new(|&_| true), vec![])
351 };
352
353 Ok((
354 Self {
355 dmap,
356 matcher,
357 root_dir,
358 options,
359 ignore_fn,
360 },
361 warnings,
362 ))
363 }
364
365 /// Is the path ignored?
366 pub fn is_ignored(&self, path: impl AsRef<HgPath>) -> bool {
367 (self.ignore_fn)(path.as_ref())
368 }
369
370 /// Is the path or one of its ancestors ignored?
371 pub fn dir_ignore(&self, dir: impl AsRef<HgPath>) -> bool {
372 // Only involve ignore mechanism if we're listing unknowns or ignored.
373 if self.options.list_ignored || self.options.list_unknown {
374 if self.is_ignored(&dir) {
375 true
376 } else {
377 for p in find_dirs(dir.as_ref()) {
378 if self.is_ignored(p) {
379 return true;
380 }
381 }
382 false
383 }
384 } else {
385 true
386 }
387 }
388
389 /// Get stat data about the files explicitly specified by the matcher.
390 /// Returns a tuple of the directories that need to be traversed and the
391 /// files with their corresponding `Dispatch`.
392 /// TODO subrepos
393 #[timed]
394 pub fn walk_explicit(
395 &self,
396 traversed_sender: crossbeam_channel::Sender<HgPathBuf>,
397 ) -> (Vec<DispatchedPath<'a>>, Vec<DispatchedPath<'a>>) {
398 self.matcher
399 .file_set()
400 .unwrap_or(&DEFAULT_WORK)
401 .par_iter()
402 .flat_map(|&filename| -> Option<_> {
403 // TODO normalization
404 let normalized = filename;
405
406 let buf = match hg_path_to_path_buf(normalized) {
407 Ok(x) => x,
408 Err(_) => {
409 return Some((
410 Cow::Borrowed(normalized),
411 INVALID_PATH_DISPATCH,
412 ))
413 }
414 };
415 let target = self.root_dir.join(buf);
416 let st = target.symlink_metadata();
417 let in_dmap = self.dmap.get(normalized);
418 match st {
419 Ok(meta) => {
420 let file_type = meta.file_type();
421 return if file_type.is_file() || file_type.is_symlink()
422 {
423 if let Some(entry) = in_dmap {
424 return Some((
425 Cow::Borrowed(normalized),
426 dispatch_found(
427 &normalized,
428 *entry,
429 HgMetadata::from_metadata(meta),
430 &self.dmap.copy_map,
431 self.options,
432 ),
433 ));
434 }
435 Some((
436 Cow::Borrowed(normalized),
437 Dispatch::Unknown,
438 ))
439 } else if file_type.is_dir() {
440 if self.options.collect_traversed_dirs {
441 traversed_sender
442 .send(normalized.to_owned())
443 .expect("receiver should outlive sender");
444 }
445 Some((
446 Cow::Borrowed(normalized),
447 Dispatch::Directory {
448 was_file: in_dmap.is_some(),
449 },
450 ))
451 } else {
452 Some((
453 Cow::Borrowed(normalized),
454 Dispatch::Bad(BadMatch::BadType(
455 // TODO do more than unknown
456 // Support for all `BadType` variant
457 // varies greatly between platforms.
458 // So far, no tests check the type and
459 // this should be good enough for most
460 // users.
461 BadType::Unknown,
462 )),
463 ))
464 };
465 }
466 Err(_) => {
467 if let Some(entry) = in_dmap {
468 return Some((
469 Cow::Borrowed(normalized),
470 dispatch_missing(entry.state()),
471 ));
472 }
473 }
474 };
475 None
476 })
477 .partition(|(_, dispatch)| match dispatch {
478 Dispatch::Directory { .. } => true,
479 _ => false,
480 })
481 }
482
483 /// Walk the working directory recursively to look for changes compared to
484 /// the current `DirstateMap`.
485 ///
486 /// This takes a mutable reference to the results to account for the
487 /// `extend` in timings
488 #[timed]
489 pub fn traverse(
490 &self,
491 path: impl AsRef<HgPath>,
492 old_results: &FastHashMap<HgPathCow<'a>, Dispatch>,
493 results: &mut Vec<DispatchedPath<'a>>,
494 traversed_sender: crossbeam_channel::Sender<HgPathBuf>,
495 ) {
496 // The traversal is done in parallel, so use a channel to gather
497 // entries. `crossbeam_channel::Sender` is `Sync`, while `mpsc::Sender`
498 // is not.
499 let (files_transmitter, files_receiver) =
500 crossbeam_channel::unbounded();
501
502 self.traverse_dir(
503 &files_transmitter,
504 path,
505 &old_results,
506 traversed_sender,
507 );
508
509 // Disconnect the channel so the receiver stops waiting
510 drop(files_transmitter);
511
512 let new_results = files_receiver
513 .into_iter()
514 .par_bridge()
515 .map(|(f, d)| (Cow::Owned(f), d));
516
517 results.par_extend(new_results);
518 }
519
520 /// Dispatch a single entry (file, folder, symlink...) found during
521 /// `traverse`. If the entry is a folder that needs to be traversed, it
522 /// will be handled in a separate thread.
523 fn handle_traversed_entry<'b>(
524 &'a self,
525 scope: &rayon::Scope<'b>,
526 files_sender: &'b crossbeam_channel::Sender<(HgPathBuf, Dispatch)>,
527 old_results: &'a FastHashMap<Cow<HgPath>, Dispatch>,
528 filename: HgPathBuf,
529 dir_entry: DirEntry,
530 traversed_sender: crossbeam_channel::Sender<HgPathBuf>,
531 ) -> IoResult<()>
532 where
533 'a: 'b,
534 {
535 let file_type = dir_entry.file_type()?;
536 let entry_option = self.dmap.get(&filename);
537
538 if filename.as_bytes() == b".hg" {
539 // Could be a directory or a symlink
540 return Ok(());
541 }
542
543 if file_type.is_dir() {
544 self.handle_traversed_dir(
545 scope,
546 files_sender,
547 old_results,
548 entry_option,
549 filename,
550 traversed_sender,
551 );
552 } else if file_type.is_file() || file_type.is_symlink() {
553 if let Some(entry) = entry_option {
554 if self.matcher.matches_everything()
555 || self.matcher.matches(&filename)
556 {
557 let metadata = dir_entry.metadata()?;
558 files_sender
559 .send((
560 filename.to_owned(),
561 dispatch_found(
562 &filename,
563 *entry,
564 HgMetadata::from_metadata(metadata),
565 &self.dmap.copy_map,
566 self.options,
567 ),
568 ))
569 .unwrap();
570 }
571 } else if (self.matcher.matches_everything()
572 || self.matcher.matches(&filename))
573 && !self.is_ignored(&filename)
574 {
575 if (self.options.list_ignored
576 || self.matcher.exact_match(&filename))
577 && self.dir_ignore(&filename)
578 {
579 if self.options.list_ignored {
580 files_sender
581 .send((filename.to_owned(), Dispatch::Ignored))
582 .unwrap();
583 }
584 } else if self.options.list_unknown {
585 files_sender
586 .send((filename.to_owned(), Dispatch::Unknown))
587 .unwrap();
588 }
589 } else if self.is_ignored(&filename) && self.options.list_ignored {
590 if self.matcher.matches(&filename) {
591 files_sender
592 .send((filename.to_owned(), Dispatch::Ignored))
593 .unwrap();
594 }
595 }
596 } else if let Some(entry) = entry_option {
597 // Used to be a file or a folder, now something else.
598 if self.matcher.matches_everything()
599 || self.matcher.matches(&filename)
600 {
601 files_sender
602 .send((
603 filename.to_owned(),
604 dispatch_missing(entry.state()),
605 ))
606 .unwrap();
607 }
608 }
609
610 Ok(())
611 }
612
613 /// A directory was found in the filesystem and needs to be traversed
614 fn handle_traversed_dir<'b>(
615 &'a self,
616 scope: &rayon::Scope<'b>,
617 files_sender: &'b crossbeam_channel::Sender<(HgPathBuf, Dispatch)>,
618 old_results: &'a FastHashMap<Cow<HgPath>, Dispatch>,
619 entry_option: Option<&'a DirstateEntry>,
620 directory: HgPathBuf,
621 traversed_sender: crossbeam_channel::Sender<HgPathBuf>,
622 ) where
623 'a: 'b,
624 {
625 scope.spawn(move |_| {
626 // Nested `if` until `rust-lang/rust#53668` is stable
627 if let Some(entry) = entry_option {
628 // Used to be a file, is now a folder
629 if self.matcher.matches_everything()
630 || self.matcher.matches(&directory)
631 {
632 files_sender
633 .send((
634 directory.to_owned(),
635 dispatch_missing(entry.state()),
636 ))
637 .unwrap();
638 }
639 }
640 // Do we need to traverse it?
641 if !self.is_ignored(&directory) || self.options.list_ignored {
642 self.traverse_dir(
643 files_sender,
644 directory,
645 &old_results,
646 traversed_sender,
647 )
648 }
649 });
650 }
651
652 /// Decides whether the directory needs to be listed, and if so handles the
653 /// entries in a separate thread.
654 fn traverse_dir(
655 &self,
656 files_sender: &crossbeam_channel::Sender<(HgPathBuf, Dispatch)>,
657 directory: impl AsRef<HgPath>,
658 old_results: &FastHashMap<Cow<HgPath>, Dispatch>,
659 traversed_sender: crossbeam_channel::Sender<HgPathBuf>,
660 ) {
661 let directory = directory.as_ref();
662
663 if self.options.collect_traversed_dirs {
664 traversed_sender
665 .send(directory.to_owned())
666 .expect("receiver should outlive sender");
667 }
668
669 let visit_entries = match self.matcher.visit_children_set(directory) {
670 VisitChildrenSet::Empty => return,
671 VisitChildrenSet::This | VisitChildrenSet::Recursive => None,
672 VisitChildrenSet::Set(set) => Some(set),
673 };
674 let buf = match hg_path_to_path_buf(directory) {
675 Ok(b) => b,
676 Err(_) => {
677 files_sender
678 .send((directory.to_owned(), INVALID_PATH_DISPATCH))
679 .expect("receiver should outlive sender");
680 return;
681 }
682 };
683 let dir_path = self.root_dir.join(buf);
684
685 let skip_dot_hg = !directory.as_bytes().is_empty();
686 let entries = match list_directory(dir_path, skip_dot_hg) {
687 Err(e) => {
688 files_sender
689 .send((directory.to_owned(), dispatch_os_error(&e)))
690 .expect("receiver should outlive sender");
691 return;
692 }
693 Ok(entries) => entries,
694 };
695
696 rayon::scope(|scope| {
697 for (filename, dir_entry) in entries {
698 if let Some(ref set) = visit_entries {
699 if !set.contains(filename.deref()) {
700 continue;
701 }
702 }
703 // TODO normalize
704 let filename = if directory.is_empty() {
705 filename.to_owned()
706 } else {
707 directory.join(&filename)
708 };
709
710 if !old_results.contains_key(filename.deref()) {
711 match self.handle_traversed_entry(
712 scope,
713 files_sender,
714 old_results,
715 filename,
716 dir_entry,
717 traversed_sender.clone(),
718 ) {
719 Err(e) => {
720 files_sender
721 .send((
722 directory.to_owned(),
723 dispatch_os_error(&e),
724 ))
725 .expect("receiver should outlive sender");
726 }
727 Ok(_) => {}
728 }
729 }
730 }
731 })
732 }
733
734 /// Add the files in the dirstate to the results.
735 ///
736 /// This takes a mutable reference to the results to account for the
737 /// `extend` in timings
738 #[timed]
739 pub fn extend_from_dmap(&self, results: &mut Vec<DispatchedPath<'a>>) {
740 results.par_extend(
741 self.dmap
742 .par_iter()
743 .filter(|(path, _)| self.matcher.matches(path))
744 .map(move |(filename, entry)| {
745 let filename: &HgPath = filename;
746 let filename_as_path = match hg_path_to_path_buf(filename)
747 {
748 Ok(f) => f,
749 Err(_) => {
750 return (
751 Cow::Borrowed(filename),
752 INVALID_PATH_DISPATCH,
753 )
754 }
755 };
756 let meta = self
757 .root_dir
758 .join(filename_as_path)
759 .symlink_metadata();
760 match meta {
761 Ok(m)
762 if !(m.file_type().is_file()
763 || m.file_type().is_symlink()) =>
764 {
765 (
766 Cow::Borrowed(filename),
767 dispatch_missing(entry.state()),
768 )
769 }
770 Ok(m) => (
771 Cow::Borrowed(filename),
772 dispatch_found(
773 filename,
774 *entry,
775 HgMetadata::from_metadata(m),
776 &self.dmap.copy_map,
777 self.options,
778 ),
779 ),
780 Err(e)
781 if e.kind() == ErrorKind::NotFound
782 || e.raw_os_error() == Some(20) =>
783 {
784 // Rust does not yet have an `ErrorKind` for
785 // `NotADirectory` (errno 20)
786 // It happens if the dirstate contains `foo/bar`
787 // and foo is not a
788 // directory
789 (
790 Cow::Borrowed(filename),
791 dispatch_missing(entry.state()),
792 )
793 }
794 Err(e) => {
795 (Cow::Borrowed(filename), dispatch_os_error(&e))
796 }
797 }
798 }),
799 );
800 }
801
802 /// Checks all files that are in the dirstate but were not found during the
803 /// working directory traversal. This means that the rest must
804 /// be either ignored, under a symlink or under a new nested repo.
805 ///
806 /// This takes a mutable reference to the results to account for the
807 /// `extend` in timings
808 #[timed]
809 pub fn handle_unknowns(&self, results: &mut Vec<DispatchedPath<'a>>) {
810 let to_visit: Vec<(&HgPath, &DirstateEntry)> =
811 if results.is_empty() && self.matcher.matches_everything() {
812 self.dmap.iter().map(|(f, e)| (f.deref(), e)).collect()
813 } else {
814 // Only convert to a hashmap if needed.
815 let old_results: FastHashMap<_, _> =
816 results.iter().cloned().collect();
817 self.dmap
818 .iter()
819 .filter_map(move |(f, e)| {
820 if !old_results.contains_key(f.deref())
821 && self.matcher.matches(f)
822 {
823 Some((f.deref(), e))
824 } else {
825 None
826 }
827 })
828 .collect()
829 };
830
831 let path_auditor = PathAuditor::new(&self.root_dir);
832
833 let new_results = to_visit.into_par_iter().filter_map(
834 |(filename, entry)| -> Option<_> {
835 // Report ignored items in the dmap as long as they are not
836 // under a symlink directory.
837 if path_auditor.check(filename) {
838 // TODO normalize for case-insensitive filesystems
839 let buf = match hg_path_to_path_buf(filename) {
840 Ok(x) => x,
841 Err(_) => {
842 return Some((
843 Cow::Owned(filename.to_owned()),
844 INVALID_PATH_DISPATCH,
845 ));
846 }
847 };
848 Some((
849 Cow::Owned(filename.to_owned()),
850 match self.root_dir.join(&buf).symlink_metadata() {
851 // File was just ignored, no links, and exists
852 Ok(meta) => {
853 let metadata = HgMetadata::from_metadata(meta);
854 dispatch_found(
855 filename,
856 *entry,
857 metadata,
858 &self.dmap.copy_map,
859 self.options,
860 )
861 }
862 // File doesn't exist
863 Err(_) => dispatch_missing(entry.state()),
864 },
865 ))
866 } else {
867 // It's either missing or under a symlink directory which
868 // we, in this case, report as missing.
869 Some((
870 Cow::Owned(filename.to_owned()),
871 dispatch_missing(entry.state()),
872 ))
873 }
874 },
875 );
876
877 results.par_extend(new_results);
878 }
879 }
880
881 #[timed]
882 pub fn build_response<'a>(
883 results: impl IntoIterator<Item = DispatchedPath<'a>>,
884 traversed: Vec<HgPathCow<'a>>,
885 ) -> DirstateStatus<'a> {
886 let mut unsure = vec![];
887 let mut modified = vec![];
888 let mut added = vec![];
889 let mut removed = vec![];
890 let mut deleted = vec![];
891 let mut clean = vec![];
892 let mut ignored = vec![];
893 let mut unknown = vec![];
894 let mut bad = vec![];
895
896 for (filename, dispatch) in results.into_iter() {
897 match dispatch {
898 Dispatch::Unknown => unknown.push(filename),
899 Dispatch::Unsure => unsure.push(filename),
900 Dispatch::Modified => modified.push(filename),
901 Dispatch::Added => added.push(filename),
902 Dispatch::Removed => removed.push(filename),
903 Dispatch::Deleted => deleted.push(filename),
904 Dispatch::Clean => clean.push(filename),
905 Dispatch::Ignored => ignored.push(filename),
906 Dispatch::None => {}
907 Dispatch::Bad(reason) => bad.push((filename, reason)),
908 Dispatch::Directory { .. } => {}
909 }
910 }
911
912 DirstateStatus {
913 modified,
914 added,
915 removed,
916 deleted,
917 clean,
918 ignored,
919 unknown,
920 bad,
921 unsure,
922 traversed,
923 dirty: false,
924 }
925 }
926
927 /// Get the status of files in the working directory.
928 ///
929 /// This is the current entry-point for `hg-core` and is realistically unusable
930 /// outside of a Python context because its arguments need to provide a lot of
931 /// information that will not be necessary in the future.
932 #[timed]
933 pub fn status<'a>(
934 dmap: &'a DirstateMap,
935 matcher: &'a (dyn Matcher + Sync),
936 root_dir: PathBuf,
937 ignore_files: Vec<PathBuf>,
938 options: StatusOptions,
939 ) -> StatusResult<(DirstateStatus<'a>, Vec<PatternFileWarning>)> {
940 let (status, warnings) =
941 Status::new(dmap, matcher, root_dir, ignore_files, options)?;
942
943 Ok((status.run()?, warnings))
944 }