tests/test-debugextensions.t

streamclone: use backgroundfilecloser (issue4889)

Closing files that have been appended to is slow on Windows/NTFS.
CloseHandle() calls on this platform often take 1-10ms - and that's
on my i7-6700K Skylake processor with a modern and fast SSD. Contrast
with other I/O operations, such as writing data, which take <100us.

This means that creating/appending thousands of files can add
significant overhead. For example, cloning mozilla-central creates
~232,000 revlog files. Assuming 1ms per CloseHandle(), that yields
232s (3:52) of wall time waiting for file closes!

The impact of this overhead can be measured most directly when applying
stream clone bundles. Applying these files is effectively uncompressing
a tar archive (read: it's very fast).

Using a RAM disk (read: no I/O wait), the difference in wall time for a
`hg debugapplystreamclonebundle` for a ~1731 MB mozilla-central bundle
between Windows and Linux from the same machine is drastic:

Linux:    ~12.8s (128MB/s)
Windows: ~352.0s (4.7MB/s)

Windows is ~27.5x slower. Yikes!

After this patch:

Linux:    ~12.8s (128MB/s)
Windows: ~102.1s (16.1MB/s)

Windows is now ~3.4x faster. Unfortunately, it is still ~8x slower than
Linux. Profiling reveals a few hot code paths that could likely be
improved. But those are for other patches.

This patch introduces test-clone-uncompressed.t because existing tests
of `clone --uncompressed` are scattered about and adding a variation for
background thread closing to e.g. test-http.t doesn't feel correct.

  $ hg debugextensions

  $ debugpath=`pwd`/extwithoutinfos.py

  $ cat > extwithoutinfos.py <<EOF
  > EOF

  $ cat >> $HGRCPATH <<EOF
  > [extensions]
  > color=
  > histedit=
  > patchbomb=
  > rebase=
  > mq=
  > ext1 = $debugpath
  > EOF

  $ hg debugextensions
  color
  ext1 (untested!)
  histedit
  mq
  patchbomb
  rebase

  $ hg debugextensions -v
  color
    location: */hgext/color.pyc (glob)
    tested with: internal
  ext1
    location: */extwithoutinfos.pyc (glob)
  histedit
    location: */hgext/histedit.pyc (glob)
    tested with: internal
  mq
    location: */hgext/mq.pyc (glob)
    tested with: internal
  patchbomb
    location: */hgext/patchbomb.pyc (glob)
    tested with: internal
  rebase
    location: */hgext/rebase.pyc (glob)
    tested with: internal

  $ hg debugextensions -Tjson | sed 's|\\\\|/|g'
  [
   {
    "buglink": "",
    "name": "color",
    "source": "*/hgext/color.pyc", (glob)
    "testedwith": "internal"
   },
   {
    "buglink": "",
    "name": "ext1",
    "source": "*/extwithoutinfos.pyc", (glob)
    "testedwith": ""
   },
   {
    "buglink": "",
    "name": "histedit",
    "source": "*/hgext/histedit.pyc", (glob)
    "testedwith": "internal"
   },
   {
    "buglink": "",
    "name": "mq",
    "source": "*/hgext/mq.pyc", (glob)
    "testedwith": "internal"
   },
   {
    "buglink": "",
    "name": "patchbomb",
    "source": "*/hgext/patchbomb.pyc", (glob)
    "testedwith": "internal"
   },
   {
    "buglink": "",
    "name": "rebase",
    "source": "*/hgext/rebase.pyc", (glob)
    "testedwith": "internal"
   }
  ]