encoding: improve handling of buggy getpreferredencoding() on Mac OS X
Prior to version 2.7, calling locale.getpreferredencoding() would
always return 'mac-roman' on Mac OS X. Previously, this was handled by
a call to locale.setlocale(). Unfortunately, Python 2.6.5 and older
have a bug where isspace() would incorrectly report True for 0x85 and
0xa0 after such a call.
In order to fix this, we replace the previous _encodingfixup mapping
to an _encodingfixers mapping. Rather than mapping encodings to their
replacement, it maps them to a function returning the
replacement. This allows us to provide an simplified implementation of
getpreferredencoding() which extracts the expected encoding and
restores the locale.
This fix is based on a patch originally submitted by Martijn Pieters
as well as feedback from Brodie Rao.
# ==== setup LF repository ====
% hg init
adding .hgeol
adding native.txt
adding unix.txt
adding win.txt
% hg clone repo repo-LF
4 files updated, 0 files merged, 0 files removed, 0 files unresolved
% printrepr.py native.txt
first
second
third
% printrepr.py unix.txt
first
second
third
% printrepr.py win.txt
first\r
second\r
third\r
% hg diff
diff --git a/native.txt b/native.txt
--- a/native.txt
+++ b/native.txt
@@ -1,3 +1,2 @@
first
-second
third
diff --git a/unix.txt b/unix.txt
--- a/unix.txt
+++ b/unix.txt
@@ -1,3 +1,2 @@
first
-second
third
diff --git a/win.txt b/win.txt
--- a/win.txt
+++ b/win.txt
@@ -1,3 +1,2 @@
first\r
-second\r
third\r
% hg revert
reverting native.txt
reverting unix.txt
reverting win.txt
% hg import
applying p
% printrepr.py native.txt
first
third
% printrepr.py unix.txt
first
third
% printrepr.py win.txt
first\r
third\r
% hg diff -c tip
diff --git a/native.txt b/native.txt
--- a/native.txt
+++ b/native.txt
@@ -1,3 +1,2 @@
first
-second
third
diff --git a/unix.txt b/unix.txt
--- a/unix.txt
+++ b/unix.txt
@@ -1,3 +1,2 @@
first
-second
third
diff --git a/win.txt b/win.txt
--- a/win.txt
+++ b/win.txt
@@ -1,3 +1,2 @@
first\r
-second\r
third\r
% hg clone repo repo-CRLF
4 files updated, 0 files merged, 0 files removed, 0 files unresolved
% printrepr.py native.txt
first\r
second\r
third\r
% printrepr.py unix.txt
first
second
third
% printrepr.py win.txt
first\r
second\r
third\r
% hg diff
diff --git a/native.txt b/native.txt
--- a/native.txt
+++ b/native.txt
@@ -1,3 +1,2 @@
first
-second
third
diff --git a/unix.txt b/unix.txt
--- a/unix.txt
+++ b/unix.txt
@@ -1,3 +1,2 @@
first
-second
third
diff --git a/win.txt b/win.txt
--- a/win.txt
+++ b/win.txt
@@ -1,3 +1,2 @@
first\r
-second\r
third\r
% hg revert
reverting native.txt
reverting unix.txt
reverting win.txt
% hg import
applying p
% printrepr.py native.txt
first\r
third\r
% printrepr.py unix.txt
first
third
% printrepr.py win.txt
first\r
third\r
% hg diff -c tip
diff --git a/native.txt b/native.txt
--- a/native.txt
+++ b/native.txt
@@ -1,3 +1,2 @@
first
-second
third
diff --git a/unix.txt b/unix.txt
--- a/unix.txt
+++ b/unix.txt
@@ -1,3 +1,2 @@
first
-second
third
diff --git a/win.txt b/win.txt
--- a/win.txt
+++ b/win.txt
@@ -1,3 +1,2 @@
first\r
-second\r
third\r
# ==== setup CRLF repository ====
% hg init
adding .hgeol
adding native.txt
adding unix.txt
adding win.txt
% hg clone repo repo-LF
4 files updated, 0 files merged, 0 files removed, 0 files unresolved
% printrepr.py native.txt
first
second
third
% printrepr.py unix.txt
first
second
third
% printrepr.py win.txt
first\r
second\r
third\r
% hg diff
diff --git a/native.txt b/native.txt
--- a/native.txt
+++ b/native.txt
@@ -1,3 +1,2 @@
first\r
-second\r
third\r
diff --git a/unix.txt b/unix.txt
--- a/unix.txt
+++ b/unix.txt
@@ -1,3 +1,2 @@
first
-second
third
diff --git a/win.txt b/win.txt
--- a/win.txt
+++ b/win.txt
@@ -1,3 +1,2 @@
first\r
-second\r
third\r
% hg revert
reverting native.txt
reverting unix.txt
reverting win.txt
% hg import
applying p
% printrepr.py native.txt
first
third
% printrepr.py unix.txt
first
third
% printrepr.py win.txt
first\r
third\r
% hg diff -c tip
diff --git a/native.txt b/native.txt
--- a/native.txt
+++ b/native.txt
@@ -1,3 +1,2 @@
first\r
-second\r
third\r
diff --git a/unix.txt b/unix.txt
--- a/unix.txt
+++ b/unix.txt
@@ -1,3 +1,2 @@
first
-second
third
diff --git a/win.txt b/win.txt
--- a/win.txt
+++ b/win.txt
@@ -1,3 +1,2 @@
first\r
-second\r
third\r
% hg clone repo repo-CRLF
4 files updated, 0 files merged, 0 files removed, 0 files unresolved
% printrepr.py native.txt
first\r
second\r
third\r
% printrepr.py unix.txt
first
second
third
% printrepr.py win.txt
first\r
second\r
third\r
% hg diff
diff --git a/native.txt b/native.txt
--- a/native.txt
+++ b/native.txt
@@ -1,3 +1,2 @@
first\r
-second\r
third\r
diff --git a/unix.txt b/unix.txt
--- a/unix.txt
+++ b/unix.txt
@@ -1,3 +1,2 @@
first
-second
third
diff --git a/win.txt b/win.txt
--- a/win.txt
+++ b/win.txt
@@ -1,3 +1,2 @@
first\r
-second\r
third\r
% hg revert
reverting native.txt
reverting unix.txt
reverting win.txt
% hg import
applying p
% printrepr.py native.txt
first\r
third\r
% printrepr.py unix.txt
first
third
% printrepr.py win.txt
first\r
third\r
% hg diff -c tip
diff --git a/native.txt b/native.txt
--- a/native.txt
+++ b/native.txt
@@ -1,3 +1,2 @@
first\r
-second\r
third\r
diff --git a/unix.txt b/unix.txt
--- a/unix.txt
+++ b/unix.txt
@@ -1,3 +1,2 @@
first
-second
third
diff --git a/win.txt b/win.txt
--- a/win.txt
+++ b/win.txt
@@ -1,3 +1,2 @@
first\r
-second\r
third\r