py3: convert an exception message to bytes
One of our users ran into this (unrelated to my recent work on
more specific errors).
Differential Revision: https://phab.mercurial-scm.org/D9184
ERROR: FIXME: This is a known failure of filelog.size for data starting with \1\nOK.