view tests/test-globalopts.t @ 35569:964212780daf

rust: implementation of `hg`

This commit provides a mostly-working implementation of the
`hg` script in Rust along with scaffolding to support Rust in
the repository.

If you are familiar with Rust, the contents of the added rust/
directory should be pretty straightforward. We create an "hgcli"
package that implements a binary application to run Mercurial.
The output of this package is an "hg" binary.

Our Rust `hg` (henceforth "rhg") essentially is a port of the existing
`hg` Python script. The main difference is the creation of the embedded
CPython interpreter is handled by the binary itself instead of relying
on the shebang. In that sense, rhg is more similar to the "exe wrapper"
we currently use on Windows. However, unlike the exe wrapper, rhg does
not call the `hg` Python script. Instead, it uses the CPython APIs to
import mercurial modules and call appropriate functions. The amount of
code here is surprisingly small.

It is my intent to replace the existing C-based exe wrapper with rhg.
Preferably in the next Mercurial release. This should be achievable -
at least for some Mercurial distributions. The future/timeline for
rhg on other platforms is less clear. We already ship a hg.exe on
Windows. So if we get the quirks with Rust worked out, shipping a
Rust-based hg.exe should hopefully not be too contentious.

Now onto the implementation.

We're using python27-sys and the cpython crates for talking to the
CPython API. We currently don't use too much functionality of the
cpython crate and could have probably cut it out. However, it does
provide a reasonable abstraction over unsafe {} CPython function
calls. While we still have our fair share of those, at least we're
not dealing with too much refcounting, error checking, etc. So I
think the use of the cpython crate is justified. Plus, there is
not-yet-implemented functionality that could benefit from cpython. I
see our use of this crate only increasing.

The cpython and python27-sys crates are not without their issues.
The cpython crate didn't seem to account for the embedding use case
in its design. Instead, it seems to assume that you are building
a Python extension. It is making some questionable decisions around
certain CPython APIs. For example, it insists that
PyEval_ThreadsInitialized() is called and that the Python code
likely isn't the main thread in the underlying application. It
is also missing some functionality that is important for embedded
use cases (such as exporting the path to the Python interpreter
from its build script). After spending several hours trying to
wrangle python27-sys and cpython, I gave up and forked the project
on GitHub. Our Cargo.toml tracks this fork. I'm optimistic that
the upstream project will accept our contributions and we can
eventually unfork.

There is a non-trivial amount of code in our custom Cargo build
script. Our build.rs (which is called as part of building the hgcli
crate):

* Validates that the Python interpreter that was detected by the
  python27-sys crate provides a shared library (we only support
  shared library linking at this time - although this restriction
  could be loosened).
* Validates that the Python is built with UCS-4 support. This ensures
  maximum Unicode compatibility.
* Exports variables to the crate build allowing the built crate to e.g.
  find the path to the Python interpreter.

The produced rhg should be considered alpha quality. There are several
known deficiencies. Many of these are documented with inline TODOs.

Probably the biggest limitation of rhg is that it assumes it is
running from the ./rust/target/<target> directory of a source
distribution. So, rhg is currently not very practical for real-world
use. But, if you can `cargo build` it, running the binary *should*
yield a working Mercurial CLI.

In order to support using rhg with the test harness, we needed to hack
up run-tests.py so the path to Mercurial's Python files is set properly.
The change is extremely hacky and is only intended to be a stop-gap
until the test harness gains first-class support for installing rhg.
This will likely occur after we support running rhg outside the
source directory.

Despite its officially alpha quality, rhg copes extremely well with
the test harness (at least on Linux). Using
`run-tests.py --with-hg ../rust/target/debug/hg`, I only encounter
the following failures:

* test-run-tests.t -- Warnings emitted about using an unexpected
  Mercurial library. This is due to the hacky nature of setting the
  Python directory when run-tests.py detected rhg.
* test-devel-warnings.t -- Expected stack trace missing frame for `hg`
  (This is expected since we no longer have an `hg` script!)
* test-convert.t -- Test running `$PYTHON "$BINDIR"/hg`, which obviously
  assumes `hg` is a Python script.
* test-merge-tools.t -- Same assumption about `hg` being executable with
  Python.
* test-http-bad-server.t -- Seeing exit code 255 instead of 1 around
  line 358.
* test-blackbox.t -- Exit code 255 instead of 1.
* test-basic.t -- Exit code 255 instead of 1.

It certainly looks like we have a bug around exit code handling. I
don't think it is severe enough to hold up review and landing of this
initial implementation. Perfect is the enemy of good.

Differential Revision: https://phab.mercurial-scm.org/D1581

  $ hg init a
  $ cd a
  $ echo a > a
  $ hg ci -A -d'1 0' -m a
  adding a

  $ cd ..

  $ hg init b
  $ cd b
  $ echo b > b
  $ hg ci -A -d'1 0' -m b
  adding b

  $ cd ..

  $ hg clone a c
  updating to branch default
  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
  $ cd c
  $ cat >> .hg/hgrc <<EOF
  > [paths]
  > relative = ../a
  > EOF
  $ hg pull -f ../b
  pulling from ../b
  searching for changes
  warning: repository is unrelated
  requesting all changes
  adding changesets
  adding manifests
  adding file changes
  added 1 changesets with 1 changes to 1 files (+1 heads)
  new changesets b6c483daf290
  (run 'hg heads' to see heads, 'hg merge' to merge)
  $ hg merge
  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
  (branch merge, don't forget to commit)

  $ cd ..

Testing -R/--repository:

  $ hg -R a tip
  changeset:   0:8580ff50825a
  tag:         tip
  user:        test
  date:        Thu Jan 01 00:00:01 1970 +0000
  summary:     a
  
  $ hg --repository b tip
  changeset:   0:b6c483daf290
  tag:         tip
  user:        test
  date:        Thu Jan 01 00:00:01 1970 +0000
  summary:     b
  

-R with a URL:

  $ hg -R file:a identify
  8580ff50825a tip
  $ hg -R file://localhost/`pwd`/a/ identify
  8580ff50825a tip

-R with path aliases:

  $ cd c
  $ hg -R default identify
  8580ff50825a tip
  $ hg -R relative identify
  8580ff50825a tip
  $ echo '[paths]' >> $HGRCPATH
  $ echo 'relativetohome = a' >> $HGRCPATH
  $ HOME=`pwd`/../ hg -R relativetohome identify
  8580ff50825a tip
  $ cd ..

#if no-outer-repo

Implicit -R:

  $ hg ann a/a
  0: a
  $ hg ann a/a a/a
  0: a
  $ hg ann a/a b/b
  abort: no repository found in '$TESTTMP' (.hg not found)!
  [255]
  $ hg -R b ann a/a
  abort: a/a not under root '$TESTTMP/b'
  (consider using '--cwd b')
  [255]
  $ hg log
  abort: no repository found in '$TESTTMP' (.hg not found)!
  [255]

#endif

Abbreviation of long option:

  $ hg --repo c tip
  changeset:   1:b6c483daf290
  tag:         tip
  parent:      -1:000000000000
  user:        test
  date:        Thu Jan 01 00:00:01 1970 +0000
  summary:     b
  

earlygetopt with duplicate options (36d23de02da1):

  $ hg --cwd a --cwd b --cwd c tip
  changeset:   1:b6c483daf290
  tag:         tip
  parent:      -1:000000000000
  user:        test
  date:        Thu Jan 01 00:00:01 1970 +0000
  summary:     b
  
  $ hg --repo c --repository b -R a tip
  changeset:   0:8580ff50825a
  tag:         tip
  user:        test
  date:        Thu Jan 01 00:00:01 1970 +0000
  summary:     a
  

earlygetopt short option without following space:

  $ hg -q -Rb tip
  0:b6c483daf290

earlygetopt with illegal abbreviations:

  $ hg --confi "foo.bar=baz"
  abort: option --config may not be abbreviated!
  [255]
  $ hg --cw a tip
  abort: option --cwd may not be abbreviated!
  [255]
  $ hg --rep a tip
  abort: option -R has to be separated from other options (e.g. not -qR) and --repository may only be abbreviated as --repo!
  [255]
  $ hg --repositor a tip
  abort: option -R has to be separated from other options (e.g. not -qR) and --repository may only be abbreviated as --repo!
  [255]
  $ hg -qR a tip
  abort: option -R has to be separated from other options (e.g. not -qR) and --repository may only be abbreviated as --repo!
  [255]
  $ hg -qRa tip
  abort: option -R has to be separated from other options (e.g. not -qR) and --repository may only be abbreviated as --repo!
  [255]

Testing --cwd:

  $ hg --cwd a parents
  changeset:   0:8580ff50825a
  tag:         tip
  user:        test
  date:        Thu Jan 01 00:00:01 1970 +0000
  summary:     a
  

Testing -y/--noninteractive - just be sure it is parsed:

  $ hg --cwd a tip -q --noninteractive
  0:8580ff50825a
  $ hg --cwd a tip -q -y
  0:8580ff50825a

Testing -q/--quiet:

  $ hg -R a -q tip
  0:8580ff50825a
  $ hg -R b -q tip
  0:b6c483daf290
  $ hg -R c --quiet parents
  0:8580ff50825a
  1:b6c483daf290

Testing -v/--verbose:

  $ hg --cwd c head -v
  changeset:   1:b6c483daf290
  tag:         tip
  parent:      -1:000000000000
  user:        test
  date:        Thu Jan 01 00:00:01 1970 +0000
  files:       b
  description:
  b
  
  
  changeset:   0:8580ff50825a
  user:        test
  date:        Thu Jan 01 00:00:01 1970 +0000
  files:       a
  description:
  a
  
  
  $ hg --cwd b tip --verbose
  changeset:   0:b6c483daf290
  tag:         tip
  user:        test
  date:        Thu Jan 01 00:00:01 1970 +0000
  files:       b
  description:
  b
  
  

Testing --config:

  $ hg --cwd c --config paths.quuxfoo=bar paths | grep quuxfoo > /dev/null && echo quuxfoo
  quuxfoo
  $ hg --cwd c --config '' tip -q
  abort: malformed --config option: '' (use --config section.name=value)
  [255]
  $ hg --cwd c --config a.b tip -q
  abort: malformed --config option: 'a.b' (use --config section.name=value)
  [255]
  $ hg --cwd c --config a tip -q
  abort: malformed --config option: 'a' (use --config section.name=value)
  [255]
  $ hg --cwd c --config a.= tip -q
  abort: malformed --config option: 'a.=' (use --config section.name=value)
  [255]
  $ hg --cwd c --config .b= tip -q
  abort: malformed --config option: '.b=' (use --config section.name=value)
  [255]

Testing --debug:

  $ hg --cwd c log --debug
  changeset:   1:b6c483daf2907ce5825c0bb50f5716226281cc1a
  tag:         tip
  phase:       public
  parent:      -1:0000000000000000000000000000000000000000
  parent:      -1:0000000000000000000000000000000000000000
  manifest:    1:23226e7a252cacdc2d99e4fbdc3653441056de49
  user:        test
  date:        Thu Jan 01 00:00:01 1970 +0000
  files+:      b
  extra:       branch=default
  description:
  b
  
  
  changeset:   0:8580ff50825a50c8f716709acdf8de0deddcd6ab
  phase:       public
  parent:      -1:0000000000000000000000000000000000000000
  parent:      -1:0000000000000000000000000000000000000000
  manifest:    0:a0c8bcbbb45c63b90b70ad007bf38961f64f2af0
  user:        test
  date:        Thu Jan 01 00:00:01 1970 +0000
  files+:      a
  extra:       branch=default
  description:
  a
  
  

Testing --traceback:

#if no-chg
  $ hg --cwd c --config x --traceback id 2>&1 | grep -i 'traceback'
  Traceback (most recent call last):
#else
Traceback for '--config' errors not supported with chg.
  $ hg --cwd c --config x --traceback id 2>&1 | grep -i 'traceback'
  [1]
#endif

Testing --time:

  $ hg --cwd a --time id
  8580ff50825a tip
  time: real * (glob)

Testing --version:

  $ hg --version -q
  Mercurial Distributed SCM * (glob)

hide outer repo
  $ hg init

Testing -h/--help:

  $ hg -h
  Mercurial Distributed SCM
  
  list of commands:
  
   add           add the specified files on the next commit
   addremove     add all new files, delete all missing files
   annotate      show changeset information by line for each file
   archive       create an unversioned archive of a repository revision
   backout       reverse effect of earlier changeset
   bisect        subdivision search of changesets
   bookmarks     create a new bookmark or list existing bookmarks
   branch        set or show the current branch name
   branches      list repository named branches
   bundle        create a bundle file
   cat           output the current or given revision of files
   clone         make a copy of an existing repository
   commit        commit the specified files or all outstanding changes
   config        show combined config settings from all hgrc files
   copy          mark files as copied for the next commit
   diff          diff repository (or selected files)
   export        dump the header and diffs for one or more changesets
   files         list tracked files
   forget        forget the specified files on the next commit
   graft         copy changes from other branches onto the current branch
   grep          search revision history for a pattern in specified files
   heads         show branch heads
   help          show help for a given topic or a help overview
   identify      identify the working directory or specified revision
   import        import an ordered set of patches
   incoming      show new changesets found in source
   init          create a new repository in the given directory
   log           show revision history of entire repository or files
   manifest      output the current or given revision of the project manifest
   merge         merge another revision into working directory
   outgoing      show changesets not found in the destination
   paths         show aliases for remote repositories
   phase         set or show the current phase name
   pull          pull changes from the specified source
   push          push changes to the specified destination
   recover       roll back an interrupted transaction
   remove        remove the specified files on the next commit
   rename        rename files; equivalent of copy + remove
   resolve       redo merges or set/view the merge status of files
   revert        restore files to their checkout state
   root          print the root (top) of the current working directory
   serve         start stand-alone webserver
   status        show changed files in the working directory
   summary       summarize working directory state
   tag           add one or more tags for the current or given revision
   tags          list repository tags
   unbundle      apply one or more bundle files
   update        update working directory (or switch revisions)
   verify        verify the integrity of the repository
   version       output version and copyright information
  
  additional help topics:
  
   bundlespec    Bundle File Formats
   color         Colorizing Outputs
   config        Configuration Files
   dates         Date Formats
   diffs         Diff Formats
   environment   Environment Variables
   extensions    Using Additional Features
   filesets      Specifying File Sets
   flags         Command-line flags
   glossary      Glossary
   hgignore      Syntax for Mercurial Ignore Files
   hgweb         Configuring hgweb
   internals     Technical implementation topics
   merge-tools   Merge Tools
   pager         Pager Support
   patterns      File Name Patterns
   phases        Working with Phases
   revisions     Specifying Revisions
   scripting     Using Mercurial from scripts and automation
   subrepos      Subrepositories
   templating    Template Usage
   urls          URL Paths
  
  (use 'hg help -v' to show built-in aliases and global options)



  $ hg --help
  Mercurial Distributed SCM
  
  list of commands:
  
   add           add the specified files on the next commit
   addremove     add all new files, delete all missing files
   annotate      show changeset information by line for each file
   archive       create an unversioned archive of a repository revision
   backout       reverse effect of earlier changeset
   bisect        subdivision search of changesets
   bookmarks     create a new bookmark or list existing bookmarks
   branch        set or show the current branch name
   branches      list repository named branches
   bundle        create a bundle file
   cat           output the current or given revision of files
   clone         make a copy of an existing repository
   commit        commit the specified files or all outstanding changes
   config        show combined config settings from all hgrc files
   copy          mark files as copied for the next commit
   diff          diff repository (or selected files)
   export        dump the header and diffs for one or more changesets
   files         list tracked files
   forget        forget the specified files on the next commit
   graft         copy changes from other branches onto the current branch
   grep          search revision history for a pattern in specified files
   heads         show branch heads
   help          show help for a given topic or a help overview
   identify      identify the working directory or specified revision
   import        import an ordered set of patches
   incoming      show new changesets found in source
   init          create a new repository in the given directory
   log           show revision history of entire repository or files
   manifest      output the current or given revision of the project manifest
   merge         merge another revision into working directory
   outgoing      show changesets not found in the destination
   paths         show aliases for remote repositories
   phase         set or show the current phase name
   pull          pull changes from the specified source
   push          push changes to the specified destination
   recover       roll back an interrupted transaction
   remove        remove the specified files on the next commit
   rename        rename files; equivalent of copy + remove
   resolve       redo merges or set/view the merge status of files
   revert        restore files to their checkout state
   root          print the root (top) of the current working directory
   serve         start stand-alone webserver
   status        show changed files in the working directory
   summary       summarize working directory state
   tag           add one or more tags for the current or given revision
   tags          list repository tags
   unbundle      apply one or more bundle files
   update        update working directory (or switch revisions)
   verify        verify the integrity of the repository
   version       output version and copyright information
  
  additional help topics:
  
   bundlespec    Bundle File Formats
   color         Colorizing Outputs
   config        Configuration Files
   dates         Date Formats
   diffs         Diff Formats
   environment   Environment Variables
   extensions    Using Additional Features
   filesets      Specifying File Sets
   flags         Command-line flags
   glossary      Glossary
   hgignore      Syntax for Mercurial Ignore Files
   hgweb         Configuring hgweb
   internals     Technical implementation topics
   merge-tools   Merge Tools
   pager         Pager Support
   patterns      File Name Patterns
   phases        Working with Phases
   revisions     Specifying Revisions
   scripting     Using Mercurial from scripts and automation
   subrepos      Subrepositories
   templating    Template Usage
   urls          URL Paths
  
  (use 'hg help -v' to show built-in aliases and global options)

Not tested: --debugger