]> git.ipfire.org Git - thirdparty/git.git/commit - utf8.c
utf8: add function to detect a missing UTF-16/32 BOM
authorLars Schneider <larsxschneider@gmail.com>
Sun, 15 Apr 2018 18:16:06 +0000 (20:16 +0200)
committerJunio C Hamano <gitster@pobox.com>
Mon, 16 Apr 2018 02:40:56 +0000 (11:40 +0900)
commitc6e48652f69f6955bbbb423100e0df2a49467db8
tree5cd147a495304941965ac2f6c2af74ec233ecd2c
parent10ecb82e4f1f507d5f122e00fd4829b30953f853
utf8: add function to detect a missing UTF-16/32 BOM

If the endianness is not defined in the encoding name, then let's
be strict and require a BOM to avoid any encoding confusion. The
is_missing_required_utf_bom() function returns true if a required BOM
is missing.

The Unicode standard instructs to assume big-endian if there in no BOM
for UTF-16/32 [1][2]. However, the W3C/WHATWG encoding standard used
in HTML5 recommends to assume little-endian to "deal with deployed
content" [3]. Strictly requiring a BOM seems to be the safest option
for content in Git.

This function is used in a subsequent commit.

[1] http://unicode.org/faq/utf_bom.html#gen6
[2] http://www.unicode.org/versions/Unicode10.0.0/ch03.pdf
     Section 3.10, D98, page 132
[3] https://encoding.spec.whatwg.org/#utf-16le

Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
utf8.c
utf8.h