Commit 1ac0ce4d authored by Junio C Hamano's avatar Junio C Hamano

Merge branch 'ls/checkout-encoding'

The new "checkout-encoding" attribute can ask Git to convert the
contents to the specified encoding when checking out to the working
tree (and the other way around when checking in).

* ls/checkout-encoding:
  convert: add round trip check based on 'core.checkRoundtripEncoding'
  convert: add tracing for 'working-tree-encoding' attribute
  convert: check for detectable errors in UTF encodings
  convert: add 'working-tree-encoding' attribute
  utf8: add function to detect a missing UTF-16/32 BOM
  utf8: add function to detect prohibited UTF-16/32 BOM
  utf8: teach same_encoding() alternative UTF encoding names
  strbuf: add a case insensitive starts_with()
  strbuf: add xstrdup_toupper()
  strbuf: remove unnecessary NUL assignment in xstrdup_tolower()
parents 7d7d051c e92d6225
......@@ -530,6 +530,12 @@ core.autocrlf::
This variable can be set to 'input',
in which case no output conversion is performed.
core.checkRoundtripEncoding::
A comma and/or whitespace separated list of encodings that Git
performs UTF-8 round trip checks on if they are used in an
`working-tree-encoding` attribute (see linkgit:gitattributes[5]).
The default value is `SHIFT-JIS`.
core.symlinks::
If false, symbolic links are checked out as small plain files that
contain the link text. linkgit:git-update-index[1] and
......
......@@ -279,6 +279,94 @@ few exceptions. Even though...
catch potential problems early, safety triggers.
`working-tree-encoding`
^^^^^^^^^^^^^^^^^^^^^^^
Git recognizes files encoded in ASCII or one of its supersets (e.g.
UTF-8, ISO-8859-1, ...) as text files. Files encoded in certain other
encodings (e.g. UTF-16) are interpreted as binary and consequently
built-in Git text processing tools (e.g. 'git diff') as well as most Git
web front ends do not visualize the contents of these files by default.
In these cases you can tell Git the encoding of a file in the working
directory with the `working-tree-encoding` attribute. If a file with this
attribute is added to Git, then Git reencodes the content from the
specified encoding to UTF-8. Finally, Git stores the UTF-8 encoded
content in its internal data structure (called "the index"). On checkout
the content is reencoded back to the specified encoding.
Please note that using the `working-tree-encoding` attribute may have a
number of pitfalls:
- Alternative Git implementations (e.g. JGit or libgit2) and older Git
versions (as of March 2018) do not support the `working-tree-encoding`
attribute. If you decide to use the `working-tree-encoding` attribute
in your repository, then it is strongly recommended to ensure that all
clients working with the repository support it.
For example, Microsoft Visual Studio resources files (`*.rc`) or
PowerShell script files (`*.ps1`) are sometimes encoded in UTF-16.
If you declare `*.ps1` as files as UTF-16 and you add `foo.ps1` with
a `working-tree-encoding` enabled Git client, then `foo.ps1` will be
stored as UTF-8 internally. A client without `working-tree-encoding`
support will checkout `foo.ps1` as UTF-8 encoded file. This will
typically cause trouble for the users of this file.
If a Git client, that does not support the `working-tree-encoding`
attribute, adds a new file `bar.ps1`, then `bar.ps1` will be
stored "as-is" internally (in this example probably as UTF-16).
A client with `working-tree-encoding` support will interpret the
internal contents as UTF-8 and try to convert it to UTF-16 on checkout.
That operation will fail and cause an error.
- Reencoding content to non-UTF encodings can cause errors as the
conversion might not be UTF-8 round trip safe. If you suspect your
encoding to not be round trip safe, then add it to
`core.checkRoundtripEncoding` to make Git check the round trip
encoding (see linkgit:git-config[1]). SHIFT-JIS (Japanese character
set) is known to have round trip issues with UTF-8 and is checked by
default.
- Reencoding content requires resources that might slow down certain
Git operations (e.g 'git checkout' or 'git add').
Use the `working-tree-encoding` attribute only if you cannot store a file
in UTF-8 encoding and if you want Git to be able to process the content
as text.
As an example, use the following attributes if your '*.ps1' files are
UTF-16 encoded with byte order mark (BOM) and you want Git to perform
automatic line ending conversion based on your platform.
------------------------
*.ps1 text working-tree-encoding=UTF-16
------------------------
Use the following attributes if your '*.ps1' files are UTF-16 little
endian encoded without BOM and you want Git to use Windows line endings
in the working directory. Please note, it is highly recommended to
explicitly define the line endings with `eol` if the `working-tree-encoding`
attribute is used to avoid ambiguity.
------------------------
*.ps1 text working-tree-encoding=UTF-16LE eol=CRLF
------------------------
You can get a list of all available encodings on your platform with the
following command:
------------------------
iconv --list
------------------------
If you do not know the encoding of a file, then you can use the `file`
command to guess the encoding:
------------------------
file foo.ps1
------------------------
`ident`
^^^^^^^
......
......@@ -1239,6 +1239,11 @@ static int git_default_core_config(const char *var, const char *value)
return 0;
}
if (!strcmp(var, "core.checkroundtripencoding")) {
check_roundtrip_encoding = xstrdup(value);
return 0;
}
if (!strcmp(var, "core.notesref")) {
notes_ref_name = xstrdup(value);
return 0;
......
This diff is collapsed.
......@@ -12,6 +12,7 @@ struct index_state;
#define CONV_EOL_RNDTRP_WARN (1<<1) /* Warn if CRLF to LF to CRLF is different */
#define CONV_EOL_RENORMALIZE (1<<2) /* Convert CRLF to LF */
#define CONV_EOL_KEEP_CRLF (1<<3) /* Keep CRLF line endings as is */
#define CONV_WRITE_OBJECT (1<<4) /* Content is written to the index */
extern int global_conv_flags_eol;
......@@ -55,6 +56,7 @@ struct delayed_checkout {
};
extern enum eol core_eol;
extern char *check_roundtrip_encoding;
extern const char *get_cached_convert_stats_ascii(const struct index_state *istate,
const char *path);
extern const char *get_wt_convert_stats_ascii(const char *path);
......
......@@ -55,6 +55,7 @@ int check_replace_refs = 1; /* NEEDSWORK: rename to read_replace_refs */
char *git_replace_ref_base;
enum eol core_eol = EOL_UNSET;
int global_conv_flags_eol = CONV_EOL_RNDTRP_WARN;
char *check_roundtrip_encoding = "SHIFT-JIS";
unsigned whitespace_rule_cfg = WS_DEFAULT_RULE;
enum branch_track git_branch_track = BRANCH_TRACK_REMOTE;
enum rebase_setup_type autorebase = AUTOREBASE_NEVER;
......
......@@ -455,6 +455,7 @@ extern void (*get_warn_routine(void))(const char *warn, va_list params);
extern void set_die_is_recursing_routine(int (*routine)(void));
extern int starts_with(const char *str, const char *prefix);
extern int istarts_with(const char *str, const char *prefix);
/*
* If the string "str" begins with the string found in "prefix", return 1.
......
......@@ -142,7 +142,7 @@ static int get_conv_flags(unsigned flags)
if (flags & HASH_RENORMALIZE)
return CONV_EOL_RENORMALIZE;
else if (flags & HASH_WRITE_OBJECT)
return global_conv_flags_eol;
return global_conv_flags_eol | CONV_WRITE_OBJECT;
else
return 0;
}
......
......@@ -11,6 +11,15 @@ int starts_with(const char *str, const char *prefix)
return 0;
}
int istarts_with(const char *str, const char *prefix)
{
for (; ; str++, prefix++)
if (!*prefix)
return 1;
else if (tolower(*str) != tolower(*prefix))
return 0;
}
int skip_to_optional_arg_default(const char *str, const char *prefix,
const char **arg, const char *def)
{
......@@ -793,7 +802,18 @@ char *xstrdup_tolower(const char *string)
result = xmallocz(len);
for (i = 0; i < len; i++)
result[i] = tolower(string[i]);
result[i] = '\0';
return result;
}
char *xstrdup_toupper(const char *string)
{
char *result;
size_t len, i;
len = strlen(string);
result = xmallocz(len);
for (i = 0; i < len; i++)
result[i] = toupper(string[i]);
return result;
}
......
......@@ -616,6 +616,7 @@ __attribute__((format (printf,2,3)))
extern int fprintf_ln(FILE *fp, const char *fmt, ...);
char *xstrdup_tolower(const char *);
char *xstrdup_toupper(const char *);
/**
* Create a newly allocated string using printf format. You can do this easily
......
#!/bin/sh
test_description='working-tree-encoding conversion via gitattributes'
. ./test-lib.sh
GIT_TRACE_WORKING_TREE_ENCODING=1 && export GIT_TRACE_WORKING_TREE_ENCODING
test_expect_success 'setup test files' '
git config core.eol lf &&
text="hallo there!\ncan you read me?" &&
echo "*.utf16 text working-tree-encoding=utf-16" >.gitattributes &&
printf "$text" >test.utf8.raw &&
printf "$text" | iconv -f UTF-8 -t UTF-16 >test.utf16.raw &&
printf "$text" | iconv -f UTF-8 -t UTF-32 >test.utf32.raw &&
# Line ending tests
printf "one\ntwo\nthree\n" >lf.utf8.raw &&
printf "one\r\ntwo\r\nthree\r\n" >crlf.utf8.raw &&
# BOM tests
printf "\0a\0b\0c" >nobom.utf16be.raw &&
printf "a\0b\0c\0" >nobom.utf16le.raw &&
printf "\376\777\0a\0b\0c" >bebom.utf16be.raw &&
printf "\777\376a\0b\0c\0" >lebom.utf16le.raw &&
printf "\0\0\0a\0\0\0b\0\0\0c" >nobom.utf32be.raw &&
printf "a\0\0\0b\0\0\0c\0\0\0" >nobom.utf32le.raw &&
printf "\0\0\376\777\0\0\0a\0\0\0b\0\0\0c" >bebom.utf32be.raw &&
printf "\777\376\0\0a\0\0\0b\0\0\0c\0\0\0" >lebom.utf32le.raw &&
# Add only UTF-16 file, we will add the UTF-32 file later
cp test.utf16.raw test.utf16 &&
cp test.utf32.raw test.utf32 &&
git add .gitattributes test.utf16 &&
git commit -m initial
'
test_expect_success 'ensure UTF-8 is stored in Git' '
test_when_finished "rm -f test.utf16.git" &&
git cat-file -p :test.utf16 >test.utf16.git &&
test_cmp_bin test.utf8.raw test.utf16.git
'
test_expect_success 're-encode to UTF-16 on checkout' '
test_when_finished "rm -f test.utf16.raw" &&
rm test.utf16 &&
git checkout test.utf16 &&
test_cmp_bin test.utf16.raw test.utf16
'
test_expect_success 'check $GIT_DIR/info/attributes support' '
test_when_finished "rm -f test.utf32.git" &&
test_when_finished "git reset --hard HEAD" &&
echo "*.utf32 text working-tree-encoding=utf-32" >.git/info/attributes &&
git add test.utf32 &&
git cat-file -p :test.utf32 >test.utf32.git &&
test_cmp_bin test.utf8.raw test.utf32.git
'
for i in 16 32
do
test_expect_success "check prohibited UTF-${i} BOM" '
test_when_finished "git reset --hard HEAD" &&
echo "*.utf${i}be text working-tree-encoding=utf-${i}be" >>.gitattributes &&
echo "*.utf${i}le text working-tree-encoding=utf-${i}LE" >>.gitattributes &&
# Here we add a UTF-16 (resp. UTF-32) files with BOM (big/little-endian)
# but we tell Git to treat it as UTF-16BE/UTF-16LE (resp. UTF-32).
# In these cases the BOM is prohibited.
cp bebom.utf${i}be.raw bebom.utf${i}be &&
test_must_fail git add bebom.utf${i}be 2>err.out &&
test_i18ngrep "fatal: BOM is prohibited .* utf-${i}be" err.out &&
test_i18ngrep "use UTF-${i} as working-tree-encoding" err.out &&
cp lebom.utf${i}le.raw lebom.utf${i}be &&
test_must_fail git add lebom.utf${i}be 2>err.out &&
test_i18ngrep "fatal: BOM is prohibited .* utf-${i}be" err.out &&
test_i18ngrep "use UTF-${i} as working-tree-encoding" err.out &&
cp bebom.utf${i}be.raw bebom.utf${i}le &&
test_must_fail git add bebom.utf${i}le 2>err.out &&
test_i18ngrep "fatal: BOM is prohibited .* utf-${i}LE" err.out &&
test_i18ngrep "use UTF-${i} as working-tree-encoding" err.out &&
cp lebom.utf${i}le.raw lebom.utf${i}le &&
test_must_fail git add lebom.utf${i}le 2>err.out &&
test_i18ngrep "fatal: BOM is prohibited .* utf-${i}LE" err.out &&
test_i18ngrep "use UTF-${i} as working-tree-encoding" err.out
'
test_expect_success "check required UTF-${i} BOM" '
test_when_finished "git reset --hard HEAD" &&
echo "*.utf${i} text working-tree-encoding=utf-${i}" >>.gitattributes &&
cp nobom.utf${i}be.raw nobom.utf${i} &&
test_must_fail git add nobom.utf${i} 2>err.out &&
test_i18ngrep "fatal: BOM is required .* utf-${i}" err.out &&
test_i18ngrep "use UTF-${i}BE or UTF-${i}LE" err.out &&
cp nobom.utf${i}le.raw nobom.utf${i} &&
test_must_fail git add nobom.utf${i} 2>err.out &&
test_i18ngrep "fatal: BOM is required .* utf-${i}" err.out &&
test_i18ngrep "use UTF-${i}BE or UTF-${i}LE" err.out
'
test_expect_success "eol conversion for UTF-${i} encoded files on checkout" '
test_when_finished "rm -f crlf.utf${i}.raw lf.utf${i}.raw" &&
test_when_finished "git reset --hard HEAD^" &&
cat lf.utf8.raw | iconv -f UTF-8 -t UTF-${i} >lf.utf${i}.raw &&
cat crlf.utf8.raw | iconv -f UTF-8 -t UTF-${i} >crlf.utf${i}.raw &&
cp crlf.utf${i}.raw eol.utf${i} &&
cat >expectIndexLF <<-EOF &&
i/lf w/-text attr/text eol.utf${i}
EOF
git add eol.utf${i} &&
git commit -m eol &&
# UTF-${i} with CRLF (Windows line endings)
rm eol.utf${i} &&
git -c core.eol=crlf checkout eol.utf${i} &&
test_cmp_bin crlf.utf${i}.raw eol.utf${i} &&
# Although the file has CRLF in the working tree,
# ensure LF in the index
git ls-files --eol eol.utf${i} >actual &&
test_cmp expectIndexLF actual &&
# UTF-${i} with LF (Unix line endings)
rm eol.utf${i} &&
git -c core.eol=lf checkout eol.utf${i} &&
test_cmp_bin lf.utf${i}.raw eol.utf${i} &&
# The file LF in the working tree, ensure LF in the index
git ls-files --eol eol.utf${i} >actual &&
test_cmp expectIndexLF actual
'
done
test_expect_success 'check unsupported encodings' '
test_when_finished "git reset --hard HEAD" &&
echo "*.set text working-tree-encoding" >.gitattributes &&
printf "set" >t.set &&
test_must_fail git add t.set 2>err.out &&
test_i18ngrep "true/false are no valid working-tree-encodings" err.out &&
echo "*.unset text -working-tree-encoding" >.gitattributes &&
printf "unset" >t.unset &&
git add t.unset &&
echo "*.empty text working-tree-encoding=" >.gitattributes &&
printf "empty" >t.empty &&
git add t.empty &&
echo "*.garbage text working-tree-encoding=garbage" >.gitattributes &&
printf "garbage" >t.garbage &&
test_must_fail git add t.garbage 2>err.out &&
test_i18ngrep "failed to encode" err.out
'
test_expect_success 'error if encoding round trip is not the same during refresh' '
BEFORE_STATE=$(git rev-parse HEAD) &&
test_when_finished "git reset --hard $BEFORE_STATE" &&
# Add and commit a UTF-16 file but skip the "working-tree-encoding"
# filter. Consequently, the in-repo representation is UTF-16 and not
# UTF-8. This simulates a Git version that has no working tree encoding
# support.
echo "*.utf16le text working-tree-encoding=utf-16le" >.gitattributes &&
echo "hallo" >nonsense.utf16le &&
TEST_HASH=$(git hash-object --no-filters -w nonsense.utf16le) &&
git update-index --add --cacheinfo 100644 $TEST_HASH nonsense.utf16le &&
COMMIT=$(git commit-tree -p $(git rev-parse HEAD) -m "plain commit" $(git write-tree)) &&
git update-ref refs/heads/master $COMMIT &&
test_must_fail git checkout HEAD^ 2>err.out &&
test_i18ngrep "error: .* overwritten by checkout:" err.out
'
test_expect_success 'error if encoding garbage is already in Git' '
BEFORE_STATE=$(git rev-parse HEAD) &&
test_when_finished "git reset --hard $BEFORE_STATE" &&
# Skip the UTF-16 filter for the added file
# This simulates a Git version that has no checkoutEncoding support
cp nobom.utf16be.raw nonsense.utf16 &&
TEST_HASH=$(git hash-object --no-filters -w nonsense.utf16) &&
git update-index --add --cacheinfo 100644 $TEST_HASH nonsense.utf16 &&
COMMIT=$(git commit-tree -p $(git rev-parse HEAD) -m "plain commit" $(git write-tree)) &&
git update-ref refs/heads/master $COMMIT &&
git diff 2>err.out &&
test_i18ngrep "error: BOM is required" err.out
'
test_expect_success 'check roundtrip encoding' '
test_when_finished "rm -f roundtrip.shift roundtrip.utf16" &&
test_when_finished "git reset --hard HEAD" &&
text="hallo there!\nroundtrip test here!" &&
printf "$text" | iconv -f UTF-8 -t SHIFT-JIS >roundtrip.shift &&
printf "$text" | iconv -f UTF-8 -t UTF-16 >roundtrip.utf16 &&
echo "*.shift text working-tree-encoding=SHIFT-JIS" >>.gitattributes &&
# SHIFT-JIS encoded files are round-trip checked by default...
GIT_TRACE=1 git add .gitattributes roundtrip.shift 2>&1 |
grep "Checking roundtrip encoding for SHIFT-JIS" &&
git reset &&
# ... unless we overwrite the Git config!
! GIT_TRACE=1 git -c core.checkRoundtripEncoding=garbage \
add .gitattributes roundtrip.shift 2>&1 |
grep "Checking roundtrip encoding for SHIFT-JIS" &&
git reset &&
# UTF-16 encoded files should not be round-trip checked by default...
! GIT_TRACE=1 git add roundtrip.utf16 2>&1 |
grep "Checking roundtrip encoding for UTF-16" &&
git reset &&
# ... unless we tell Git to check it!
GIT_TRACE=1 git -c core.checkRoundtripEncoding="UTF-16, UTF-32" \
add roundtrip.utf16 2>&1 |
grep "Checking roundtrip encoding for utf-16" &&
git reset &&
# ... unless we tell Git to check it!
# (here we also check that the casing of the encoding is irrelevant)
GIT_TRACE=1 git -c core.checkRoundtripEncoding="UTF-32, utf-16" \
add roundtrip.utf16 2>&1 |
grep "Checking roundtrip encoding for utf-16" &&
git reset
'
test_done
......@@ -401,18 +401,40 @@ void strbuf_utf8_replace(struct strbuf *sb_src, int pos, int width,
strbuf_release(&sb_dst);
}
/*
* Returns true (1) if the src encoding name matches the dst encoding
* name directly or one of its alternative names. E.g. UTF-16BE is the
* same as UTF16BE.
*/
static int same_utf_encoding(const char *src, const char *dst)
{
if (istarts_with(src, "utf") && istarts_with(dst, "utf")) {
/* src[3] or dst[3] might be '\0' */
int i = (src[3] == '-' ? 4 : 3);
int j = (dst[3] == '-' ? 4 : 3);
return !strcasecmp(src+i, dst+j);
}
return 0;
}
int is_encoding_utf8(const char *name)
{
if (!name)
return 1;
if (!strcasecmp(name, "utf-8") || !strcasecmp(name, "utf8"))
if (same_utf_encoding("utf-8", name))
return 1;
return 0;
}
int same_encoding(const char *src, const char *dst)
{
if (is_encoding_utf8(src) && is_encoding_utf8(dst))
static const char utf8[] = "UTF-8";
if (!src)
src = utf8;
if (!dst)
dst = utf8;
if (same_utf_encoding(src, dst))
return 1;
return !strcasecmp(src, dst);
}
......@@ -538,6 +560,45 @@ char *reencode_string_len(const char *in, int insz,
}
#endif
static int has_bom_prefix(const char *data, size_t len,
const char *bom, size_t bom_len)
{
return data && bom && (len >= bom_len) && !memcmp(data, bom, bom_len);
}
static const char utf16_be_bom[] = {0xFE, 0xFF};
static const char utf16_le_bom[] = {0xFF, 0xFE};
static const char utf32_be_bom[] = {0x00, 0x00, 0xFE, 0xFF};
static const char utf32_le_bom[] = {0xFF, 0xFE, 0x00, 0x00};
int has_prohibited_utf_bom(const char *enc, const char *data, size_t len)
{
return (
(same_utf_encoding("UTF-16BE", enc) ||
same_utf_encoding("UTF-16LE", enc)) &&
(has_bom_prefix(data, len, utf16_be_bom, sizeof(utf16_be_bom)) ||
has_bom_prefix(data, len, utf16_le_bom, sizeof(utf16_le_bom)))
) || (
(same_utf_encoding("UTF-32BE", enc) ||
same_utf_encoding("UTF-32LE", enc)) &&
(has_bom_prefix(data, len, utf32_be_bom, sizeof(utf32_be_bom)) ||
has_bom_prefix(data, len, utf32_le_bom, sizeof(utf32_le_bom)))
);
}
int is_missing_required_utf_bom(const char *enc, const char *data, size_t len)
{
return (
(same_utf_encoding(enc, "UTF-16")) &&
!(has_bom_prefix(data, len, utf16_be_bom, sizeof(utf16_be_bom)) ||
has_bom_prefix(data, len, utf16_le_bom, sizeof(utf16_le_bom)))
) || (
(same_utf_encoding(enc, "UTF-32")) &&
!(has_bom_prefix(data, len, utf32_be_bom, sizeof(utf32_be_bom)) ||
has_bom_prefix(data, len, utf32_le_bom, sizeof(utf32_le_bom)))
);
}
/*
* Returns first character length in bytes for multi-byte `text` according to
* `encoding`.
......
......@@ -70,4 +70,32 @@ typedef enum {
void strbuf_utf8_align(struct strbuf *buf, align_type position, unsigned int width,
const char *s);
/*
* If a data stream is declared as UTF-16BE or UTF-16LE, then a UTF-16
* BOM must not be used [1]. The same applies for the UTF-32 equivalents.
* The function returns true if this rule is violated.
*
* [1] http://unicode.org/faq/utf_bom.html#bom10
*/
int has_prohibited_utf_bom(const char *enc, const char *data, size_t len);
/*
* If the endianness is not defined in the encoding name, then we
* require a BOM. The function returns true if a required BOM is missing.
*
* The Unicode standard instructs to assume big-endian if there in no
* BOM for UTF-16/32 [1][2]. However, the W3C/WHATWG encoding standard
* used in HTML5 recommends to assume little-endian to "deal with
* deployed content" [3].
*
* Therefore, strictly requiring a BOM seems to be the safest option for
* content in Git.
*
* [1] http://unicode.org/faq/utf_bom.html#gen6
* [2] http://www.unicode.org/versions/Unicode10.0.0/ch03.pdf
* Section 3.10, D98, page 132
* [3] https://encoding.spec.whatwg.org/#utf-16le
*/
int is_missing_required_utf_bom(const char *enc, const char *data, size_t len);
#endif
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment