Rework UTF-16 conversion functions.
- rename strncpy_from_utf16be to strncat_from_utf16be and its feature is slightly changed as strcat.
- rename strncpy_to_utf16be to strncat_to_utf16be and its feature is slightly changed as strcat.
- use U+FFFD for invalid character if destination charset is UTF-8.
- proper handle surrogate pair on Windows version.
Rework a handling of memory allocation failure in string conversion;
report the failure to the caller as much as we can instead of calling __archive_errx().
Clean up best_effort_strncat_in_locale().
- merge is_all_ascii_code() and best_effort_strncat_to_utf8() into best_effort_strncat_in_locale().
- assign '?' character if a character code > 127.
- update comment.
The source data type for ConvertFromUnicodeToText() should be UniChar, UTF-16
so we have to do pre-conversion that translates a UTF-8 string to UTF-16's
before calling ConvertFromUnicodeToText() and pass that UTF-16 string.
Improve MBS<=>WCS functions.
- split archive_wstring_append_from_mbs into mbsnrtowcs version and mbrtowc/mbtowc version.
- POSIX version of archive_string_append_from_wcs behavior simulates Windows version's.
That is to continue conversion even if there is illegal WC.
- set the terminator '\0' to destination buffer even if conversion failed.
Use archive_mstring_*_l interface in archive_acl_* functions
since archive_mstring knows how to treat UTF-8 characters and
what way is efficient on the platform and what way reduces or
avoid dependence on current locale.
Increase robustness for an empty filename or an invalid filename on Windows,
which cannot be completely converted from WCS to MBS.
Correctly detect invalid UTF-8 string on Windows.
Specify -fno-builtin to gcc during detecting functions because use of
-Werror on CMake prevent the detection of built-in functions of gcc by
compiling failure which is conflicting types for built-in function.
Implement a string conversion interface to archive_entry and archive_mstring
for efficient string conversion. Some platform have to do a string conversion
through wide characters. And then Windows platform cannot make locale UTF-8,
so it means use of wide characters is only way to make a internationalization
program.
Issue 106.
Check the result of FIEMAP because there is a possibility the result
has adjacent extents though the file is not sparse. So we should
clear the sparse data if it indicates the whole file.
If the character-set of filenames in archives is UTF-8, we should automatically
normalize it to avoid the scene that two filenames in one directory are, of course,
different byte sequence but they have the same sight because of NFD and NFC.
Second reason is that iconv cannot correctly convert NFD characters to other
character-set so we have to convert NFD to NFC before iconv handle it unless
iconv supports UTF-8-MAC. Third reason is for matching filenames, if filenames
in archives are NFD and the platform is not MAC OS, the uses cannot specify
the filename the uses want to extract although the users can see the filename
by listing. Recently NFD can be displayed on some platforms but creating NFD
requires character-set conversion utility, in particular that input of NFD
string is hard on Windows platform.
Tim Kientzle [Tue, 26 Apr 2011 06:02:43 +0000 (02:02 -0400)]
Refactor the read_open() routines into a collection of
single setters to set the callbacks, and a simple
archive_read_open1(a) that uses the callbacks already
registered.
In particular, this will make it easier to extend the
API in the future with new callbacks.
Get rid of a hdrcharset option support from xar reader. It is almost useless for xar
since the options would be used only when the users needed to convert filenames
from UTF-8-MAC to UTF-8, but, unfortunately, UTF-8-MAC is not usually supported
by iconv/libiconv(including packages system such as FreeBSD ports or NetBSD pkgsrc)
except MAC OS, while the patch for libiconv has been available for many years.
That means users will not use UTF-8-MAC unless the users could build libiconv with
custom libiconv for UTF-8-MAC.
Use en_US.UTF-8 instead of de_DE.UTF-8 at test_entry and test_pax_filename_encoding
because it seems en_US.UTF-8 is widely supported by default more than de_DE.UTF-8.
For example, Ubuntu has installed "C" and en_*.UTF-8 locales whatever locale would
be selected by default but de_DE.UTF-8 is not.
Improve Unicode handling.
1. The conversion will fail when following conditions since those are not legal Unicode.
- The code point larger than 0x10FFFF.
- The code point consist of overlong sequence.
- There is a surrogate pair in UTF-8 strings. we currently use iconv
if available, and it does not allow a surrogate pair in UTF-8, so the behavior
of string_append_from_utf8_to_utf16be() function should match it.
- There is a incomplete surrogate pair in UTF-16 strings.
2. Use a table for getting bytes of UTF-8 sequence from first byte.
It is easy to know what code is wrong and how many bytes are following.
Introudce "tar:utf8type=libarchive2x" option for the incorrect UTF-8 string
which libarchive 2.x makes in wrong assumption about wchar_t. The option works
only for pax format.
Currently libarchive 3 correctly translates UTF-8 string from/to current locale
string, but we cannot accordingly handle the incorrect UTF-8 on the some
platforms wchar_t of which is not Unicode and users are not using UTF-8 locale.
So we should support the UTF-8 string to be properly translated to current
locale string.
projects / thirdparty / libarchive.git / log
