Mercurial > hg
diff rust/hg-core/src/dirstate/status.rs @ 48068:bf8837e3d7ce
dirstate: Remove the flat Rust DirstateMap implementation
Before this changeset we had two Rust implementations of `DirstateMap`.
This removes the "flat" DirstateMap so that the "tree" DirstateMap is always
used when Rust enabled. This simplifies the code a lot, and will enable
(in the next changeset) further removal of a trait abstraction.
This is a performance regression when:
* Rust is enabled, and
* The repository uses the legacy dirstate-v1 file format, and
* For `hg status`, unknown files are not listed (such as with `-mard`)
The regression is about 100 milliseconds for `hg status -mard` on a
semi-large repository (mozilla-central), from ~320ms to ~420ms.
We deem this to be small enough to be worth it.
The new dirstate-v2 is still experimental at this point, but we aim to
stabilize it (though not yet enable it by default for new repositories)
in Mercurial 6.0. Eventually, upgrating repositories to dirsate-v2 will
eliminate this regression (and enable other performance improvements).
# Background
The flat DirstateMap was introduced with the first Rust implementation of the
status algorithm. It works similarly to the previous Python + C one, with a
single `HashMap` that associates file paths to a `DirstateEntry` (where Python
has a dict).
We later added the tree DirstateMap where the root of the tree contains nodes
for files and directories that are directly at the root of the repository,
and nodes for directories can contain child nodes representing the files and
directly that *they* contain directly. The shape of this tree mirrors that of
the working directory in the filesystem. This enables the status algorithm to
traverse this tree in tandem with traversing the filesystem tree, which in
turns enables a more efficient algorithm.
Furthermore, the new dirstate-v2 file format is also based on a tree of the
same shape. The tree DirstateMap can access a dirstate-v2 file without parsing
it: binary data in a single large (possibly memory-mapped) bytes buffer is
traversed on demand. This allows `DirstateMap` creation to take `O(1)` time.
(Mutation works by creating new in-memory nodes with copy-on-write semantics,
and serialization is append-mostly.)
The tradeoff is that for "legacy" repositories that use the dirstate-v1 file
format, parsing that file into a tree DirstateMap takes more time. Profiling
shows that this time is dominated by `HashMap`. For a dirstate containing `F`
files with an average `D` directory depth, the flat DirstateMap does parsing
in `O(F)` number of HashMap operations but the tree DirstateMap in `O(F × D)`
operations, since each node has its own HashMap containing its child nodes.
This slower costs ~140ms on an old snapshot of mozilla-central, and ~80ms
on an old snapshot of the Netbeans repository.
The status algorithm is faster, but with `-mard` (when not listing unknown
files) it is typically not faster *enough* to compensate the slower parsing.
Both Rust implementations are always faster than the Python + C implementation
# Benchmark results
All benchmarks are run on changeset 98c0408324e6, with repositories that use
the dirstate-v1 file format, on a server with 4 CPU cores and 4 CPU threads
(no HyperThreading).
`hg status` benchmarks show wall clock times of the entire command as the
average and standard deviation of serveral runs, collected by
https://github.com/sharkdp/hyperfine and reformated.
Parsing benchmarks are wall clock time of the Rust function that converts a
bytes buffer of the dirstate file into the `DirstateMap` data structure as
used by the status algorithm. A single run each, collected by running
`hg status` this environment variable:
RUST_LOG=hg::dirstate::dirstate_map=trace,hg::dirstate_tree::dirstate_map=trace
Benchmark 1: Rust flat DirstateMap → Rust tree DirstateMap
hg status
mozilla-clean 562.3 ms ± 2.0 ms → 462.5 ms ± 0.6 ms 1.22 ± 0.00 times faster
mozilla-dirty 859.6 ms ± 2.2 ms → 719.5 ms ± 3.2 ms 1.19 ± 0.01 times faster
mozilla-ignored 558.2 ms ± 3.0 ms → 457.9 ms ± 2.9 ms 1.22 ± 0.01 times faster
mozilla-unknowns 859.4 ms ± 5.7 ms → 716.0 ms ± 4.7 ms 1.20 ± 0.01 times faster
netbeans-clean 336.5 ms ± 0.9 ms → 339.5 ms ± 0.4 ms 0.99 ± 0.00 times faster
netbeans-dirty 491.4 ms ± 1.6 ms → 475.1 ms ± 1.2 ms 1.03 ± 0.00 times faster
netbeans-ignored 343.7 ms ± 1.0 ms → 347.8 ms ± 0.4 ms 0.99 ± 0.00 times faster
netbeans-unknowns 484.3 ms ± 1.0 ms → 466.0 ms ± 1.2 ms 1.04 ± 0.00 times faster
hg status -mard
mozilla-clean 317.3 ms ± 0.6 ms → 422.5 ms ± 1.2 ms 0.75 ± 0.00 times faster
mozilla-dirty 315.4 ms ± 0.6 ms → 417.7 ms ± 1.1 ms 0.76 ± 0.00 times faster
mozilla-ignored 314.6 ms ± 0.6 ms → 417.4 ms ± 1.0 ms 0.75 ± 0.00 times faster
mozilla-unknowns 312.9 ms ± 0.9 ms → 417.3 ms ± 1.6 ms 0.75 ± 0.00 times faster
netbeans-clean 212.0 ms ± 0.6 ms → 283.6 ms ± 0.8 ms 0.75 ± 0.00 times faster
netbeans-dirty 211.4 ms ± 1.0 ms → 283.4 ms ± 1.6 ms 0.75 ± 0.01 times faster
netbeans-ignored 211.4 ms ± 0.9 ms → 283.9 ms ± 0.8 ms 0.74 ± 0.01 times faster
netbeans-unknowns 211.1 ms ± 0.6 ms → 283.4 ms ± 1.0 ms 0.74 ± 0.00 times faster
Parsing
mozilla-clean 38.4ms → 177.6ms
mozilla-dirty 38.8ms → 177.0ms
mozilla-ignored 38.8ms → 178.0ms
mozilla-unknowns 38.7ms → 176.9ms
netbeans-clean 16.5ms → 97.3ms
netbeans-dirty 16.5ms → 98.4ms
netbeans-ignored 16.9ms → 97.4ms
netbeans-unknowns 16.9ms → 96.3ms
Benchmark 2: Python + C dirstatemap → Rust tree DirstateMap
hg status
mozilla-clean 1261.0 ms ± 3.6 ms → 461.1 ms ± 0.5 ms 2.73 ± 0.00 times faster
mozilla-dirty 2293.4 ms ± 9.1 ms → 719.6 ms ± 3.6 ms 3.19 ± 0.01 times faster
mozilla-ignored 1240.4 ms ± 2.3 ms → 457.7 ms ± 1.9 ms 2.71 ± 0.00 times faster
mozilla-unknowns 2283.3 ms ± 9.0 ms → 719.7 ms ± 3.8 ms 3.17 ± 0.01 times faster
netbeans-clean 879.7 ms ± 3.5 ms → 339.9 ms ± 0.5 ms 2.59 ± 0.00 times faster
netbeans-dirty 1257.3 ms ± 4.7 ms → 474.6 ms ± 1.6 ms 2.65 ± 0.01 times faster
netbeans-ignored 943.9 ms ± 1.9 ms → 347.3 ms ± 1.1 ms 2.72 ± 0.00 times faster
netbeans-unknowns 1188.1 ms ± 5.0 ms → 465.2 ms ± 2.3 ms 2.55 ± 0.01 times faster
hg status -mard
mozilla-clean 903.2 ms ± 3.6 ms → 423.4 ms ± 2.2 ms 2.13 ± 0.01 times faster
mozilla-dirty 884.6 ms ± 4.5 ms → 417.3 ms ± 1.4 ms 2.12 ± 0.01 times faster
mozilla-ignored 881.9 ms ± 1.3 ms → 417.3 ms ± 0.8 ms 2.11 ± 0.00 times faster
mozilla-unknowns 878.5 ms ± 1.9 ms → 416.4 ms ± 0.9 ms 2.11 ± 0.00 times faster
netbeans-clean 434.9 ms ± 1.8 ms → 284.0 ms ± 0.8 ms 1.53 ± 0.01 times faster
netbeans-dirty 434.1 ms ± 0.8 ms → 283.1 ms ± 0.8 ms 1.53 ± 0.00 times faster
netbeans-ignored 431.7 ms ± 1.1 ms → 283.6 ms ± 1.8 ms 1.52 ± 0.01 times faster
netbeans-unknowns 433.0 ms ± 1.3 ms → 283.5 ms ± 0.7 ms 1.53 ± 0.00 times faster
Differential Revision: https://phab.mercurial-scm.org/D11516
| author | Simon Sapin <simon.sapin@octobus.net> |
|---|---|
| date | Mon, 27 Sep 2021 12:09:15 +0200 |
| parents | 1b2ee68e85f9 |
| children | 269ff8978086 |
line wrap: on
line diff
--- a/rust/hg-core/src/dirstate/status.rs Tue Sep 28 20:00:19 2021 +0200 +++ b/rust/hg-core/src/dirstate/status.rs Mon Sep 27 12:09:15 2021 +0200 @@ -10,33 +10,13 @@ //! and will only be triggered in narrow cases. use crate::dirstate_tree::on_disk::DirstateV2ParseError; -use crate::utils::path_auditor::PathAuditor; + use crate::{ - dirstate::SIZE_FROM_OTHER_PARENT, - filepatterns::PatternFileWarning, - matchers::{get_ignore_function, Matcher, VisitChildrenSet}, - utils::{ - files::{find_dirs, HgMetadata}, - hg_path::{ - hg_path_to_path_buf, os_string_to_hg_path_buf, HgPath, HgPathBuf, - HgPathError, - }, - }, - CopyMap, DirstateEntry, DirstateMap, EntryState, FastHashMap, + utils::hg_path::{HgPath, HgPathError}, PatternError, }; -use lazy_static::lazy_static; -use micro_timer::timed; -use rayon::prelude::*; -use std::{ - borrow::Cow, - collections::HashSet, - fmt, - fs::{read_dir, DirEntry}, - io::ErrorKind, - ops::Deref, - path::{Path, PathBuf}, -}; + +use std::{borrow::Cow, fmt}; /// Wrong type of file from a `BadMatch` /// Note: a lot of those don't exist on all platforms. @@ -70,32 +50,6 @@ BadType(BadType), } -/// Enum used to dispatch new status entries into the right collections. -/// Is similar to `crate::EntryState`, but represents the transient state of -/// entries during the lifetime of a command. -#[derive(Debug, Copy, Clone)] -pub enum Dispatch { - Unsure, - Modified, - Added, - Removed, - Deleted, - Clean, - Unknown, - Ignored, - /// Empty dispatch, the file is not worth listing - None, - /// Was explicitly matched but cannot be found/accessed - Bad(BadMatch), - Directory { - /// True if the directory used to be a file in the dmap so we can say - /// that it's been removed. - was_file: bool, - }, -} - -type IoResult<T> = std::io::Result<T>; - /// `Box<dyn Trait>` is syntactic sugar for `Box<dyn Trait + 'static>`, so add /// an explicit lifetime here to not fight `'static` bounds "out of nowhere". pub type IgnoreFnType<'a> = @@ -105,135 +59,6 @@ /// the dirstate/explicit) paths, this comes up a lot. pub type HgPathCow<'a> = Cow<'a, HgPath>; -/// A path with its computed ``Dispatch`` information -type DispatchedPath<'a> = (HgPathCow<'a>, Dispatch); - -/// The conversion from `HgPath` to a real fs path failed. -/// `22` is the error code for "Invalid argument" -const INVALID_PATH_DISPATCH: Dispatch = Dispatch::Bad(BadMatch::OsError(22)); - -/// Dates and times that are outside the 31-bit signed range are compared -/// modulo 2^31. This should prevent hg from behaving badly with very large -/// files or corrupt dates while still having a high probability of detecting -/// changes. (issue2608) -/// TODO I haven't found a way of having `b` be `Into<i32>`, since `From<u64>` -/// is not defined for `i32`, and there is no `As` trait. This forces the -/// caller to cast `b` as `i32`. -fn mod_compare(a: i32, b: i32) -> bool { - a & i32::max_value() != b & i32::max_value() -} - -/// Return a sorted list containing information about the entries -/// in the directory. -/// -/// * `skip_dot_hg` - Return an empty vec if `path` contains a `.hg` directory -fn list_directory( - path: impl AsRef<Path>, - skip_dot_hg: bool, -) -> std::io::Result<Vec<(HgPathBuf, DirEntry)>> { - let mut results = vec![]; - let entries = read_dir(path.as_ref())?; - - for entry in entries { - let entry = entry?; - let filename = os_string_to_hg_path_buf(entry.file_name())?; - let file_type = entry.file_type()?; - if skip_dot_hg && filename.as_bytes() == b".hg" && file_type.is_dir() { - return Ok(vec![]); - } else { - results.push((filename, entry)) - } - } - - results.sort_unstable_by_key(|e| e.0.clone()); - Ok(results) -} - -/// The file corresponding to the dirstate entry was found on the filesystem. -fn dispatch_found( - filename: impl AsRef<HgPath>, - entry: DirstateEntry, - metadata: HgMetadata, - copy_map: &CopyMap, - options: StatusOptions, -) -> Dispatch { - match entry.state() { - EntryState::Normal => { - let mode = entry.mode(); - let size = entry.size(); - let mtime = entry.mtime(); - - let HgMetadata { - st_mode, - st_size, - st_mtime, - .. - } = metadata; - - let size_changed = mod_compare(size, st_size as i32); - let mode_changed = - (mode ^ st_mode as i32) & 0o100 != 0o000 && options.check_exec; - let metadata_changed = size >= 0 && (size_changed || mode_changed); - let other_parent = size == SIZE_FROM_OTHER_PARENT; - - if metadata_changed - || other_parent - || copy_map.contains_key(filename.as_ref()) - { - if metadata.is_symlink() && size_changed { - // issue6456: Size returned may be longer due to encryption - // on EXT-4 fscrypt. TODO maybe only do it on EXT4? - Dispatch::Unsure - } else { - Dispatch::Modified - } - } else if mod_compare(mtime, st_mtime as i32) - || st_mtime == options.last_normal_time - { - // the file may have just been marked as normal and - // it may have changed in the same second without - // changing its size. This can happen if we quickly - // do multiple commits. Force lookup, so we don't - // miss such a racy file change. - Dispatch::Unsure - } else if options.list_clean { - Dispatch::Clean - } else { - Dispatch::None - } - } - EntryState::Merged => Dispatch::Modified, - EntryState::Added => Dispatch::Added, - EntryState::Removed => Dispatch::Removed, - } -} - -/// The file corresponding to this Dirstate entry is missing. -fn dispatch_missing(state: EntryState) -> Dispatch { - match state { - // File was removed from the filesystem during commands - EntryState::Normal | EntryState::Merged | EntryState::Added => { - Dispatch::Deleted - } - // File was removed, everything is normal - EntryState::Removed => Dispatch::Removed, - } -} - -fn dispatch_os_error(e: &std::io::Error) -> Dispatch { - Dispatch::Bad(BadMatch::OsError( - e.raw_os_error().expect("expected real OS error"), - )) -} - -lazy_static! { - static ref DEFAULT_WORK: HashSet<&'static HgPath> = { - let mut h = HashSet::new(); - h.insert(HgPath::new(b"")); - h - }; -} - #[derive(Debug, Copy, Clone)] pub struct StatusOptions { /// Remember the most recent modification timeslot for status, to make @@ -319,626 +144,3 @@ } } } - -/// Gives information about which files are changed in the working directory -/// and how, compared to the revision we're based on -pub struct Status<'a, M: ?Sized + Matcher + Sync> { - dmap: &'a DirstateMap, - pub(crate) matcher: &'a M, - root_dir: PathBuf, - pub(crate) options: StatusOptions, - ignore_fn: IgnoreFnType<'a>, -} - -impl<'a, M> Status<'a, M> -where - M: ?Sized + Matcher + Sync, -{ - pub fn new( - dmap: &'a DirstateMap, - matcher: &'a M, - root_dir: PathBuf, - ignore_files: Vec<PathBuf>, - options: StatusOptions, - ) -> StatusResult<(Self, Vec<PatternFileWarning>)> { - // Needs to outlive `dir_ignore_fn` since it's captured. - - let (ignore_fn, warnings): (IgnoreFnType, _) = - if options.list_ignored || options.list_unknown { - get_ignore_function(ignore_files, &root_dir, &mut |_| {})? - } else { - (Box::new(|&_| true), vec![]) - }; - - Ok(( - Self { - dmap, - matcher, - root_dir, - options, - ignore_fn, - }, - warnings, - )) - } - - /// Is the path ignored? - pub fn is_ignored(&self, path: impl AsRef<HgPath>) -> bool { - (self.ignore_fn)(path.as_ref()) - } - - /// Is the path or one of its ancestors ignored? - pub fn dir_ignore(&self, dir: impl AsRef<HgPath>) -> bool { - // Only involve ignore mechanism if we're listing unknowns or ignored. - if self.options.list_ignored || self.options.list_unknown { - if self.is_ignored(&dir) { - true - } else { - for p in find_dirs(dir.as_ref()) { - if self.is_ignored(p) { - return true; - } - } - false - } - } else { - true - } - } - - /// Get stat data about the files explicitly specified by the matcher. - /// Returns a tuple of the directories that need to be traversed and the - /// files with their corresponding `Dispatch`. - /// TODO subrepos - #[timed] - pub fn walk_explicit( - &self, - traversed_sender: crossbeam_channel::Sender<HgPathBuf>, - ) -> (Vec<DispatchedPath<'a>>, Vec<DispatchedPath<'a>>) { - self.matcher - .file_set() - .unwrap_or(&DEFAULT_WORK) - .par_iter() - .flat_map(|&filename| -> Option<_> { - // TODO normalization - let normalized = filename; - - let buf = match hg_path_to_path_buf(normalized) { - Ok(x) => x, - Err(_) => { - return Some(( - Cow::Borrowed(normalized), - INVALID_PATH_DISPATCH, - )) - } - }; - let target = self.root_dir.join(buf); - let st = target.symlink_metadata(); - let in_dmap = self.dmap.get(normalized); - match st { - Ok(meta) => { - let file_type = meta.file_type(); - return if file_type.is_file() || file_type.is_symlink() - { - if let Some(entry) = in_dmap { - return Some(( - Cow::Borrowed(normalized), - dispatch_found( - &normalized, - *entry, - HgMetadata::from_metadata(meta), - &self.dmap.copy_map, - self.options, - ), - )); - } - Some(( - Cow::Borrowed(normalized), - Dispatch::Unknown, - )) - } else if file_type.is_dir() { - if self.options.collect_traversed_dirs { - traversed_sender - .send(normalized.to_owned()) - .expect("receiver should outlive sender"); - } - Some(( - Cow::Borrowed(normalized), - Dispatch::Directory { - was_file: in_dmap.is_some(), - }, - )) - } else { - Some(( - Cow::Borrowed(normalized), - Dispatch::Bad(BadMatch::BadType( - // TODO do more than unknown - // Support for all `BadType` variant - // varies greatly between platforms. - // So far, no tests check the type and - // this should be good enough for most - // users. - BadType::Unknown, - )), - )) - }; - } - Err(_) => { - if let Some(entry) = in_dmap { - return Some(( - Cow::Borrowed(normalized), - dispatch_missing(entry.state()), - )); - } - } - }; - None - }) - .partition(|(_, dispatch)| match dispatch { - Dispatch::Directory { .. } => true, - _ => false, - }) - } - - /// Walk the working directory recursively to look for changes compared to - /// the current `DirstateMap`. - /// - /// This takes a mutable reference to the results to account for the - /// `extend` in timings - #[timed] - pub fn traverse( - &self, - path: impl AsRef<HgPath>, - old_results: &FastHashMap<HgPathCow<'a>, Dispatch>, - results: &mut Vec<DispatchedPath<'a>>, - traversed_sender: crossbeam_channel::Sender<HgPathBuf>, - ) { - // The traversal is done in parallel, so use a channel to gather - // entries. `crossbeam_channel::Sender` is `Sync`, while `mpsc::Sender` - // is not. - let (files_transmitter, files_receiver) = - crossbeam_channel::unbounded(); - - self.traverse_dir( - &files_transmitter, - path, - &old_results, - traversed_sender, - ); - - // Disconnect the channel so the receiver stops waiting - drop(files_transmitter); - - let new_results = files_receiver - .into_iter() - .par_bridge() - .map(|(f, d)| (Cow::Owned(f), d)); - - results.par_extend(new_results); - } - - /// Dispatch a single entry (file, folder, symlink...) found during - /// `traverse`. If the entry is a folder that needs to be traversed, it - /// will be handled in a separate thread. - fn handle_traversed_entry<'b>( - &'a self, - scope: &rayon::Scope<'b>, - files_sender: &'b crossbeam_channel::Sender<(HgPathBuf, Dispatch)>, - old_results: &'a FastHashMap<Cow<HgPath>, Dispatch>, - filename: HgPathBuf, - dir_entry: DirEntry, - traversed_sender: crossbeam_channel::Sender<HgPathBuf>, - ) -> IoResult<()> - where - 'a: 'b, - { - let file_type = dir_entry.file_type()?; - let entry_option = self.dmap.get(&filename); - - if filename.as_bytes() == b".hg" { - // Could be a directory or a symlink - return Ok(()); - } - - if file_type.is_dir() { - self.handle_traversed_dir( - scope, - files_sender, - old_results, - entry_option, - filename, - traversed_sender, - ); - } else if file_type.is_file() || file_type.is_symlink() { - if let Some(entry) = entry_option { - if self.matcher.matches_everything() - || self.matcher.matches(&filename) - { - let metadata = dir_entry.metadata()?; - files_sender - .send(( - filename.to_owned(), - dispatch_found( - &filename, - *entry, - HgMetadata::from_metadata(metadata), - &self.dmap.copy_map, - self.options, - ), - )) - .unwrap(); - } - } else if (self.matcher.matches_everything() - || self.matcher.matches(&filename)) - && !self.is_ignored(&filename) - { - if (self.options.list_ignored - || self.matcher.exact_match(&filename)) - && self.dir_ignore(&filename) - { - if self.options.list_ignored { - files_sender - .send((filename.to_owned(), Dispatch::Ignored)) - .unwrap(); - } - } else if self.options.list_unknown { - files_sender - .send((filename.to_owned(), Dispatch::Unknown)) - .unwrap(); - } - } else if self.is_ignored(&filename) && self.options.list_ignored { - if self.matcher.matches(&filename) { - files_sender - .send((filename.to_owned(), Dispatch::Ignored)) - .unwrap(); - } - } - } else if let Some(entry) = entry_option { - // Used to be a file or a folder, now something else. - if self.matcher.matches_everything() - || self.matcher.matches(&filename) - { - files_sender - .send(( - filename.to_owned(), - dispatch_missing(entry.state()), - )) - .unwrap(); - } - } - - Ok(()) - } - - /// A directory was found in the filesystem and needs to be traversed - fn handle_traversed_dir<'b>( - &'a self, - scope: &rayon::Scope<'b>, - files_sender: &'b crossbeam_channel::Sender<(HgPathBuf, Dispatch)>, - old_results: &'a FastHashMap<Cow<HgPath>, Dispatch>, - entry_option: Option<&'a DirstateEntry>, - directory: HgPathBuf, - traversed_sender: crossbeam_channel::Sender<HgPathBuf>, - ) where - 'a: 'b, - { - scope.spawn(move |_| { - // Nested `if` until `rust-lang/rust#53668` is stable - if let Some(entry) = entry_option { - // Used to be a file, is now a folder - if self.matcher.matches_everything() - || self.matcher.matches(&directory) - { - files_sender - .send(( - directory.to_owned(), - dispatch_missing(entry.state()), - )) - .unwrap(); - } - } - // Do we need to traverse it? - if !self.is_ignored(&directory) || self.options.list_ignored { - self.traverse_dir( - files_sender, - directory, - &old_results, - traversed_sender, - ) - } - }); - } - - /// Decides whether the directory needs to be listed, and if so handles the - /// entries in a separate thread. - fn traverse_dir( - &self, - files_sender: &crossbeam_channel::Sender<(HgPathBuf, Dispatch)>, - directory: impl AsRef<HgPath>, - old_results: &FastHashMap<Cow<HgPath>, Dispatch>, - traversed_sender: crossbeam_channel::Sender<HgPathBuf>, - ) { - let directory = directory.as_ref(); - - if self.options.collect_traversed_dirs { - traversed_sender - .send(directory.to_owned()) - .expect("receiver should outlive sender"); - } - - let visit_entries = match self.matcher.visit_children_set(directory) { - VisitChildrenSet::Empty => return, - VisitChildrenSet::This | VisitChildrenSet::Recursive => None, - VisitChildrenSet::Set(set) => Some(set), - }; - let buf = match hg_path_to_path_buf(directory) { - Ok(b) => b, - Err(_) => { - files_sender - .send((directory.to_owned(), INVALID_PATH_DISPATCH)) - .expect("receiver should outlive sender"); - return; - } - }; - let dir_path = self.root_dir.join(buf); - - let skip_dot_hg = !directory.as_bytes().is_empty(); - let entries = match list_directory(dir_path, skip_dot_hg) { - Err(e) => { - files_sender - .send((directory.to_owned(), dispatch_os_error(&e))) - .expect("receiver should outlive sender"); - return; - } - Ok(entries) => entries, - }; - - rayon::scope(|scope| { - for (filename, dir_entry) in entries { - if let Some(ref set) = visit_entries { - if !set.contains(filename.deref()) { - continue; - } - } - // TODO normalize - let filename = if directory.is_empty() { - filename.to_owned() - } else { - directory.join(&filename) - }; - - if !old_results.contains_key(filename.deref()) { - match self.handle_traversed_entry( - scope, - files_sender, - old_results, - filename, - dir_entry, - traversed_sender.clone(), - ) { - Err(e) => { - files_sender - .send(( - directory.to_owned(), - dispatch_os_error(&e), - )) - .expect("receiver should outlive sender"); - } - Ok(_) => {} - } - } - } - }) - } - - /// Add the files in the dirstate to the results. - /// - /// This takes a mutable reference to the results to account for the - /// `extend` in timings - #[timed] - pub fn extend_from_dmap(&self, results: &mut Vec<DispatchedPath<'a>>) { - results.par_extend( - self.dmap - .par_iter() - .filter(|(path, _)| self.matcher.matches(path)) - .map(move |(filename, entry)| { - let filename: &HgPath = filename; - let filename_as_path = match hg_path_to_path_buf(filename) - { - Ok(f) => f, - Err(_) => { - return ( - Cow::Borrowed(filename), - INVALID_PATH_DISPATCH, - ) - } - }; - let meta = self - .root_dir - .join(filename_as_path) - .symlink_metadata(); - match meta { - Ok(m) - if !(m.file_type().is_file() - || m.file_type().is_symlink()) => - { - ( - Cow::Borrowed(filename), - dispatch_missing(entry.state()), - ) - } - Ok(m) => ( - Cow::Borrowed(filename), - dispatch_found( - filename, - *entry, - HgMetadata::from_metadata(m), - &self.dmap.copy_map, - self.options, - ), - ), - Err(e) - if e.kind() == ErrorKind::NotFound - || e.raw_os_error() == Some(20) => - { - // Rust does not yet have an `ErrorKind` for - // `NotADirectory` (errno 20) - // It happens if the dirstate contains `foo/bar` - // and foo is not a - // directory - ( - Cow::Borrowed(filename), - dispatch_missing(entry.state()), - ) - } - Err(e) => { - (Cow::Borrowed(filename), dispatch_os_error(&e)) - } - } - }), - ); - } - - /// Checks all files that are in the dirstate but were not found during the - /// working directory traversal. This means that the rest must - /// be either ignored, under a symlink or under a new nested repo. - /// - /// This takes a mutable reference to the results to account for the - /// `extend` in timings - #[timed] - pub fn handle_unknowns(&self, results: &mut Vec<DispatchedPath<'a>>) { - let to_visit: Vec<(&HgPath, &DirstateEntry)> = - if results.is_empty() && self.matcher.matches_everything() { - self.dmap.iter().map(|(f, e)| (f.deref(), e)).collect() - } else { - // Only convert to a hashmap if needed. - let old_results: FastHashMap<_, _> = - results.iter().cloned().collect(); - self.dmap - .iter() - .filter_map(move |(f, e)| { - if !old_results.contains_key(f.deref()) - && self.matcher.matches(f) - { - Some((f.deref(), e)) - } else { - None - } - }) - .collect() - }; - - let path_auditor = PathAuditor::new(&self.root_dir); - - let new_results = to_visit.into_par_iter().filter_map( - |(filename, entry)| -> Option<_> { - // Report ignored items in the dmap as long as they are not - // under a symlink directory. - if path_auditor.check(filename) { - // TODO normalize for case-insensitive filesystems - let buf = match hg_path_to_path_buf(filename) { - Ok(x) => x, - Err(_) => { - return Some(( - Cow::Owned(filename.to_owned()), - INVALID_PATH_DISPATCH, - )); - } - }; - Some(( - Cow::Owned(filename.to_owned()), - match self.root_dir.join(&buf).symlink_metadata() { - // File was just ignored, no links, and exists - Ok(meta) => { - let metadata = HgMetadata::from_metadata(meta); - dispatch_found( - filename, - *entry, - metadata, - &self.dmap.copy_map, - self.options, - ) - } - // File doesn't exist - Err(_) => dispatch_missing(entry.state()), - }, - )) - } else { - // It's either missing or under a symlink directory which - // we, in this case, report as missing. - Some(( - Cow::Owned(filename.to_owned()), - dispatch_missing(entry.state()), - )) - } - }, - ); - - results.par_extend(new_results); - } -} - -#[timed] -pub fn build_response<'a>( - results: impl IntoIterator<Item = DispatchedPath<'a>>, - traversed: Vec<HgPathCow<'a>>, -) -> DirstateStatus<'a> { - let mut unsure = vec![]; - let mut modified = vec![]; - let mut added = vec![]; - let mut removed = vec![]; - let mut deleted = vec![]; - let mut clean = vec![]; - let mut ignored = vec![]; - let mut unknown = vec![]; - let mut bad = vec![]; - - for (filename, dispatch) in results.into_iter() { - match dispatch { - Dispatch::Unknown => unknown.push(filename), - Dispatch::Unsure => unsure.push(filename), - Dispatch::Modified => modified.push(filename), - Dispatch::Added => added.push(filename), - Dispatch::Removed => removed.push(filename), - Dispatch::Deleted => deleted.push(filename), - Dispatch::Clean => clean.push(filename), - Dispatch::Ignored => ignored.push(filename), - Dispatch::None => {} - Dispatch::Bad(reason) => bad.push((filename, reason)), - Dispatch::Directory { .. } => {} - } - } - - DirstateStatus { - modified, - added, - removed, - deleted, - clean, - ignored, - unknown, - bad, - unsure, - traversed, - dirty: false, - } -} - -/// Get the status of files in the working directory. -/// -/// This is the current entry-point for `hg-core` and is realistically unusable -/// outside of a Python context because its arguments need to provide a lot of -/// information that will not be necessary in the future. -#[timed] -pub fn status<'a>( - dmap: &'a DirstateMap, - matcher: &'a (dyn Matcher + Sync), - root_dir: PathBuf, - ignore_files: Vec<PathBuf>, - options: StatusOptions, -) -> StatusResult<(DirstateStatus<'a>, Vec<PatternFileWarning>)> { - let (status, warnings) = - Status::new(dmap, matcher, root_dir, ignore_files, options)?; - - Ok((status.run()?, warnings)) -}