• Torsten Bögershausen's avatar
    Support working-tree-encoding "UTF-16LE-BOM" · aab2a1ae
    Torsten Bögershausen authored
    Users who want UTF-16 files in the working tree set the .gitattributes
    like this:
    test.txt working-tree-encoding=UTF-16
    
    The unicode standard itself defines 3 allowed ways how to encode UTF-16.
    The following 3 versions convert all back to 'g' 'i' 't' in UTF-8:
    
    a) UTF-16, without BOM, big endian:
    $ printf "\000g\000i\000t" | iconv -f UTF-16 -t UTF-8 | od -c
    0000000    g   i   t
    
    b) UTF-16, with BOM, little endian:
    $ printf "\377\376g\000i\000t\000" | iconv -f UTF-16 -t UTF-8 | od -c
    0000000    g   i   t
    
    c) UTF-16, with BOM, big endian:
    $ printf "\376\377\000g\000i\000t" | iconv -f UTF-16 -t UTF-8 | od -c
    0000000    g   i   t
    
    Git uses libiconv to convert from UTF-8 in the index into ITF-16 in the
    working tree.
    After a checkout, the resulting file has a BOM and is encoded in "UTF-16",
    in the version (c) above.
    This is what iconv generates, more details follow below.
    
    iconv (and libiconv) can generate UTF-16, UTF-16LE or UTF-16BE:
    
    d) UTF-16
    $ printf 'git' | iconv -f UTF-8 -t UTF-16 | od -c
    0000000  376 377  \0   g  \0   i  \0   t
    
    e) UTF-16LE
    $ printf 'git' | iconv -f UTF-8 -t UTF-16LE | od -c
    0000000    g  \0   i  \0   t  \0
    
    f)  UTF-16BE
    $ printf 'git' | iconv -f UTF-8 -t UTF-16BE | od -c
    0000000   \0   g  \0   i  \0   t
    
    There is no way to generate version (b) from above in a Git working tree,
    but that is what some applications need.
    (All fully unicode aware applications should be able to read all 3 variants,
    but in practise we are not there yet).
    
    When producing UTF-16 as an output, iconv generates the big endian version
    with a BOM. (big endian is probably chosen for historical reasons).
    
    iconv can produce UTF-16 files with little endianess by using "UTF-16LE"
    as encoding, and that file does not have a BOM.
    
    Not all users (especially under Windows) are happy with this.
    Some tools are not fully unicode aware and can only handle version (b).
    
    Today there is no way to produce version (b) with iconv (or libiconv).
    Looking into the history of iconv, it seems as if version (c) will
    be used in all future iconv versions (for compatibility reasons).
    
    Solve this dilemma and introduce a Git-specific "UTF-16LE-BOM".
    libiconv can not handle the encoding, so Git pick it up, handles the BOM
    and uses libiconv to convert the rest of the stream.
    (UTF-16BE-BOM is added for consistency)
    Rported-by: Adrián's avatarAdrián Gimeno Balaguer <adrigibal@gmail.com>
    Signed-off-by: default avatarTorsten Bögershausen <tboegi@web.de>
    Signed-off-by: default avatarJunio C Hamano <gitster@pobox.com>
    aab2a1ae
Name
Last commit
Last update
.github Loading commit data...
Documentation Loading commit data...
block-sha1 Loading commit data...
builtin Loading commit data...
ci Loading commit data...
compat Loading commit data...
contrib Loading commit data...
ewah Loading commit data...
git-gui Loading commit data...
gitk-git Loading commit data...
gitweb Loading commit data...
mergetools Loading commit data...
negotiator Loading commit data...
perl Loading commit data...
po Loading commit data...
ppc Loading commit data...
refs Loading commit data...
sha1collisiondetection @ 232357eb Loading commit data...
sha1dc Loading commit data...
t Loading commit data...
templates Loading commit data...
vcs-svn Loading commit data...
xdiff Loading commit data...
.clang-format Loading commit data...
.editorconfig Loading commit data...
.gitattributes Loading commit data...
.gitignore Loading commit data...
.gitmodules Loading commit data...
.mailmap Loading commit data...
.travis.yml Loading commit data...
.tsan-suppressions Loading commit data...
COPYING Loading commit data...
GIT-VERSION-GEN Loading commit data...
INSTALL Loading commit data...
LGPL-2.1 Loading commit data...
Makefile Loading commit data...
README.md Loading commit data...
RelNotes Loading commit data...
abspath.c Loading commit data...
aclocal.m4 Loading commit data...
advice.c Loading commit data...
advice.h Loading commit data...
alias.c Loading commit data...
alias.h Loading commit data...
alloc.c Loading commit data...
alloc.h Loading commit data...
apply.c Loading commit data...
apply.h Loading commit data...
archive-tar.c Loading commit data...
archive-zip.c Loading commit data...
archive.c Loading commit data...
archive.h Loading commit data...
argv-array.c Loading commit data...
argv-array.h Loading commit data...
attr.c Loading commit data...
attr.h Loading commit data...
banned.h Loading commit data...
base85.c Loading commit data...
bisect.c Loading commit data...
bisect.h Loading commit data...
blame.c Loading commit data...
blame.h Loading commit data...
blob.c Loading commit data...
blob.h Loading commit data...
branch.c Loading commit data...
branch.h Loading commit data...
builtin.h Loading commit data...
bulk-checkin.c Loading commit data...
bulk-checkin.h Loading commit data...
bundle.c Loading commit data...
bundle.h Loading commit data...
cache-tree.c Loading commit data...
cache-tree.h Loading commit data...
cache.h Loading commit data...
chdir-notify.c Loading commit data...
chdir-notify.h Loading commit data...
check-builtins.sh Loading commit data...
check-racy.c Loading commit data...
check_bindir Loading commit data...
checkout.c Loading commit data...
checkout.h Loading commit data...
color.c Loading commit data...
color.h Loading commit data...
column.c Loading commit data...
column.h Loading commit data...
combine-diff.c Loading commit data...
command-list.txt Loading commit data...
commit-graph.c Loading commit data...
commit-graph.h Loading commit data...
commit-reach.c Loading commit data...
commit-reach.h Loading commit data...
commit-slab-decl.h Loading commit data...
commit-slab-impl.h Loading commit data...
commit-slab.h Loading commit data...
commit.c Loading commit data...
commit.h Loading commit data...
common-main.c Loading commit data...
config.c Loading commit data...
config.h Loading commit data...
config.mak.dev Loading commit data...
config.mak.in Loading commit data...
config.mak.uname Loading commit data...
configure.ac Loading commit data...
connect.c Loading commit data...
connect.h Loading commit data...
connected.c Loading commit data...
connected.h Loading commit data...
convert.c Loading commit data...
convert.h Loading commit data...
copy.c Loading commit data...
credential-cache--daemon.c Loading commit data...
credential-cache.c Loading commit data...
credential-store.c Loading commit data...
credential.c Loading commit data...
credential.h Loading commit data...
csum-file.c Loading commit data...
csum-file.h Loading commit data...
ctype.c Loading commit data...
daemon.c Loading commit data...
date.c Loading commit data...
decorate.c Loading commit data...
decorate.h Loading commit data...
delta-islands.c Loading commit data...
delta-islands.h Loading commit data...
delta.h Loading commit data...
detect-compiler Loading commit data...
diff-delta.c Loading commit data...
diff-lib.c Loading commit data...
diff-no-index.c Loading commit data...
diff.c Loading commit data...
diff.h Loading commit data...
diffcore-break.c Loading commit data...
diffcore-delta.c Loading commit data...
diffcore-order.c Loading commit data...
diffcore-pickaxe.c Loading commit data...
diffcore-rename.c Loading commit data...
diffcore.h Loading commit data...
dir-iterator.c Loading commit data...
dir-iterator.h Loading commit data...
dir.c Loading commit data...
dir.h Loading commit data...
editor.c Loading commit data...
entry.c Loading commit data...
environment.c Loading commit data...
exec-cmd.c Loading commit data...
exec-cmd.h Loading commit data...
fast-import.c Loading commit data...
fetch-negotiator.c Loading commit data...
fetch-negotiator.h Loading commit data...
fetch-object.c Loading commit data...
fetch-object.h Loading commit data...
fetch-pack.c Loading commit data...
fetch-pack.h Loading commit data...
fmt-merge-msg.h Loading commit data...
fsck.c Loading commit data...
fsck.h Loading commit data...
fsmonitor.c Loading commit data...
fsmonitor.h Loading commit data...
fuzz-pack-headers.c Loading commit data...
fuzz-pack-idx.c Loading commit data...
generate-cmdlist.sh Loading commit data...
gettext.c Loading commit data...
gettext.h Loading commit data...
git-add--interactive.perl Loading commit data...
git-archimport.perl Loading commit data...
git-bisect.sh Loading commit data...
git-compat-util.h Loading commit data...
git-cvsexportcommit.perl Loading commit data...
git-cvsimport.perl Loading commit data...
git-cvsserver.perl Loading commit data...
git-difftool--helper.sh Loading commit data...
git-filter-branch.sh Loading commit data...
git-instaweb.sh Loading commit data...
git-legacy-rebase.sh Loading commit data...
git-merge-octopus.sh Loading commit data...
git-merge-one-file.sh Loading commit data...
git-merge-resolve.sh Loading commit data...
git-mergetool--lib.sh Loading commit data...
git-mergetool.sh Loading commit data...
git-p4.py Loading commit data...
git-parse-remote.sh Loading commit data...
git-quiltimport.sh Loading commit data...
git-rebase--am.sh Loading commit data...
git-rebase--common.sh Loading commit data...
git-rebase--merge.sh Loading commit data...
git-rebase--preserve-merges.sh Loading commit data...
git-remote-testgit.sh Loading commit data...
git-request-pull.sh Loading commit data...
git-send-email.perl Loading commit data...
git-sh-i18n.sh Loading commit data...
git-sh-setup.sh Loading commit data...
git-stash.sh Loading commit data...
git-submodule.sh Loading commit data...
git-svn.perl Loading commit data...
git-web--browse.sh Loading commit data...
git.c Loading commit data...
git.rc Loading commit data...
gpg-interface.c Loading commit data...
gpg-interface.h Loading commit data...
graph.c Loading commit data...
graph.h Loading commit data...
grep.c Loading commit data...
grep.h Loading commit data...
hash.h Loading commit data...
hashmap.c Loading commit data...
hashmap.h Loading commit data...
help.c Loading commit data...
help.h Loading commit data...
hex.c Loading commit data...
http-backend.c Loading commit data...
http-fetch.c Loading commit data...
http-push.c Loading commit data...
http-walker.c Loading commit data...
http.c Loading commit data...
http.h Loading commit data...
ident.c Loading commit data...
imap-send.c Loading commit data...
interdiff.c Loading commit data...
interdiff.h Loading commit data...
iterator.h Loading commit data...
json-writer.c Loading commit data...
json-writer.h Loading commit data...
khash.h Loading commit data...
kwset.c Loading commit data...
kwset.h Loading commit data...
levenshtein.c Loading commit data...
levenshtein.h Loading commit data...
line-log.c Loading commit data...
line-log.h Loading commit data...
line-range.c Loading commit data...
line-range.h Loading commit data...
linear-assignment.c Loading commit data...
linear-assignment.h Loading commit data...
list-objects-filter-options.c Loading commit data...
list-objects-filter-options.h Loading commit data...
list-objects-filter.c Loading commit data...
list-objects-filter.h Loading commit data...
list-objects.c Loading commit data...
list-objects.h Loading commit data...
list.h Loading commit data...
ll-merge.c Loading commit data...
ll-merge.h Loading commit data...
lockfile.c Loading commit data...
lockfile.h Loading commit data...
log-tree.c Loading commit data...
log-tree.h Loading commit data...
ls-refs.c Loading commit data...
ls-refs.h Loading commit data...
mailinfo.c Loading commit data...
mailinfo.h Loading commit data...
mailmap.c Loading commit data...
mailmap.h Loading commit data...
match-trees.c Loading commit data...
mem-pool.c Loading commit data...
mem-pool.h Loading commit data...
merge-blobs.c Loading commit data...
merge-blobs.h Loading commit data...
merge-recursive.c Loading commit data...
merge-recursive.h Loading commit data...
merge.c Loading commit data...
mergesort.c Loading commit data...
mergesort.h Loading commit data...
midx.c Loading commit data...
midx.h Loading commit data...
name-hash.c Loading commit data...
notes-cache.c Loading commit data...
notes-cache.h Loading commit data...
notes-merge.c Loading commit data...
notes-merge.h Loading commit data...
notes-utils.c Loading commit data...
notes-utils.h Loading commit data...
notes.c Loading commit data...
notes.h Loading commit data...
object-store.h Loading commit data...
object.c Loading commit data...
object.h Loading commit data...
oidmap.c Loading commit data...
oidmap.h Loading commit data...
oidset.c Loading commit data...
oidset.h Loading commit data...
pack-bitmap-write.c Loading commit data...
pack-bitmap.c Loading commit data...
pack-bitmap.h Loading commit data...
pack-check.c Loading commit data...
pack-objects.c Loading commit data...
pack-objects.h Loading commit data...
pack-revindex.c Loading commit data...
pack-revindex.h Loading commit data...
pack-write.c Loading commit data...
pack.h Loading commit data...
packfile.c Loading commit data...
packfile.h Loading commit data...
pager.c Loading commit data...
parse-options-cb.c Loading commit data...
parse-options.c Loading commit data...
parse-options.h Loading commit data...
patch-delta.c Loading commit data...
patch-ids.c Loading commit data...
patch-ids.h Loading commit data...
path.c Loading commit data...
path.h Loading commit data...
pathspec.c Loading commit data...
pathspec.h Loading commit data...
pkt-line.c Loading commit data...
pkt-line.h Loading commit data...
preload-index.c Loading commit data...
pretty.c Loading commit data...
pretty.h Loading commit data...
prio-queue.c Loading commit data...
prio-queue.h Loading commit data...
progress.c Loading commit data...
progress.h Loading commit data...
prompt.c Loading commit data...
prompt.h Loading commit data...
protocol.c Loading commit data...
protocol.h Loading commit data...
quote.c Loading commit data...
quote.h Loading commit data...
range-diff.c Loading commit data...
range-diff.h Loading commit data...
reachable.c Loading commit data...
reachable.h Loading commit data...
read-cache.c Loading commit data...
rebase-interactive.c Loading commit data...
rebase-interactive.h Loading commit data...
ref-filter.c Loading commit data...
ref-filter.h Loading commit data...
reflog-walk.c Loading commit data...
reflog-walk.h Loading commit data...
refs.c Loading commit data...
refs.h Loading commit data...
refspec.c Loading commit data...
refspec.h Loading commit data...
remote-curl.c Loading commit data...
remote-testsvn.c Loading commit data...
remote.c Loading commit data...
remote.h Loading commit data...
replace-object.c Loading commit data...
replace-object.h Loading commit data...
repository.c Loading commit data...
repository.h Loading commit data...
rerere.c Loading commit data...
rerere.h Loading commit data...
resolve-undo.c Loading commit data...
resolve-undo.h Loading commit data...
revision.c Loading commit data...
revision.h Loading commit data...
run-command.c Loading commit data...
run-command.h Loading commit data...
send-pack.c Loading commit data...
send-pack.h Loading commit data...
sequencer.c Loading commit data...
sequencer.h Loading commit data...
serve.c Loading commit data...
serve.h Loading commit data...
server-info.c Loading commit data...
setup.c Loading commit data...
sh-i18n--envsubst.c Loading commit data...
sha1-array.c Loading commit data...
sha1-array.h Loading commit data...
sha1-file.c Loading commit data...
sha1-lookup.c Loading commit data...
sha1-lookup.h Loading commit data...
sha1-name.c Loading commit data...
sha1dc_git.c Loading commit data...
sha1dc_git.h Loading commit data...
shallow.c Loading commit data...
shell.c Loading commit data...
shortlog.h Loading commit data...
sideband.c Loading commit data...
sideband.h Loading commit data...
sigchain.c Loading commit data...
sigchain.h Loading commit data...
split-index.c Loading commit data...
split-index.h Loading commit data...
strbuf.c Loading commit data...
strbuf.h Loading commit data...
streaming.c Loading commit data...
streaming.h Loading commit data...
string-list.c Loading commit data...
string-list.h Loading commit data...
sub-process.c Loading commit data...
sub-process.h Loading commit data...
submodule-config.c Loading commit data...
submodule-config.h Loading commit data...
submodule.c Loading commit data...
submodule.h Loading commit data...
symlinks.c Loading commit data...
tag.c Loading commit data...
tag.h Loading commit data...
tar.h Loading commit data...
tempfile.c Loading commit data...
tempfile.h Loading commit data...
thread-utils.c Loading commit data...
thread-utils.h Loading commit data...
tmp-objdir.c Loading commit data...
tmp-objdir.h Loading commit data...
trace.c Loading commit data...
trace.h Loading commit data...
trailer.c Loading commit data...
trailer.h Loading commit data...
transport-helper.c Loading commit data...
transport-internal.h Loading commit data...
transport.c Loading commit data...
transport.h Loading commit data...
tree-diff.c Loading commit data...
tree-walk.c Loading commit data...
tree-walk.h Loading commit data...
tree.c Loading commit data...
tree.h Loading commit data...
unicode-width.h Loading commit data...
unimplemented.sh Loading commit data...
unix-socket.c Loading commit data...
unix-socket.h Loading commit data...
unpack-trees.c Loading commit data...
unpack-trees.h Loading commit data...
upload-pack.c Loading commit data...
upload-pack.h Loading commit data...
url.c Loading commit data...
url.h Loading commit data...
urlmatch.c Loading commit data...
urlmatch.h Loading commit data...
usage.c Loading commit data...
userdiff.c Loading commit data...
userdiff.h Loading commit data...
utf8.c Loading commit data...
utf8.h Loading commit data...
varint.c Loading commit data...
varint.h Loading commit data...
version.c Loading commit data...
version.h Loading commit data...
versioncmp.c Loading commit data...
walker.c Loading commit data...
walker.h Loading commit data...
wildmatch.c Loading commit data...
wildmatch.h Loading commit data...
worktree.c Loading commit data...
worktree.h Loading commit data...
wrap-for-bin.sh Loading commit data...
wrapper.c Loading commit data...
write-or-die.c Loading commit data...
ws.c Loading commit data...
wt-status.c Loading commit data...
wt-status.h Loading commit data...
xdiff-interface.c Loading commit data...
xdiff-interface.h Loading commit data...
zlib.c Loading commit data...