]>
Commit | Line | Data |
---|---|---|
390955cb UD |
1 | @node Character Set Handling, Locales, String and Array Utilities, Top |
2 | @c %MENU% Support for extended character sets | |
3 | @chapter Character Set Handling | |
4 | ||
5 | @ifnottex | |
6 | @macro cal{text} | |
7 | \text\ | |
8 | @end macro | |
9 | @end ifnottex | |
10 | ||
d731df03 UD |
11 | Character sets used in the early days of computing had only six, seven, |
12 | or eight bits for each character: there was never a case where more than | |
13 | eight bits (one byte) were used to represent a single character. The | |
14 | limitations of this approach became more apparent as more people | |
15 | grappled with non-Roman character sets, where not all the characters | |
16 | that make up a language's character set can be represented by @math{2^8} | |
17 | choices. This chapter shows the functionality which was added to the C | |
aaca11d8 | 18 | library to support multiple character sets. |
390955cb UD |
19 | |
20 | @menu | |
21 | * Extended Char Intro:: Introduction to Extended Characters. | |
22 | * Charset Function Overview:: Overview about Character Handling | |
23 | Functions. | |
24 | * Restartable multibyte conversion:: Restartable multibyte conversion | |
25 | Functions. | |
26 | * Non-reentrant Conversion:: Non-reentrant Conversion Function. | |
27 | * Generic Charset Conversion:: Generic Charset Conversion. | |
28 | @end menu | |
29 | ||
30 | ||
31 | @node Extended Char Intro | |
32 | @section Introduction to Extended Characters | |
33 | ||
d731df03 UD |
34 | A variety of solutions to overcome the differences between |
35 | character sets with a 1:1 relation between bytes and characters and | |
36 | character sets with ratios of 2:1 or 4:1 exist. The remainder of this | |
37 | section gives a few examples to help understand the design decisions | |
38 | made while developing the functionality of the @w{C library}. | |
390955cb UD |
39 | |
40 | @cindex internal representation | |
41 | A distinction we have to make right away is between internal and | |
42 | external representation. @dfn{Internal representation} means the | |
43 | representation used by a program while keeping the text in memory. | |
44 | External representations are used when text is stored or transmitted | |
d731df03 UD |
45 | through whatever communication channel. Examples of external |
46 | representations include files lying in a directory that are going to be | |
47 | read and parsed. | |
390955cb | 48 | |
aaca11d8 UD |
49 | Traditionally there has been no difference between the two representations. |
50 | It was equally comfortable and useful to use the same single-byte | |
390955cb UD |
51 | representation internally and externally. This changes with more and |
52 | larger character sets. | |
53 | ||
54 | One of the problems to overcome with the internal representation is | |
aaca11d8 | 55 | handling text that is externally encoded using different character |
390955cb UD |
56 | sets. Assume a program which reads two texts and compares them using |
57 | some metric. The comparison can be usefully done only if the texts are | |
58 | internally kept in a common format. | |
59 | ||
60 | @cindex wide character | |
61 | For such a common format (@math{=} character set) eight bits are certainly | |
d731df03 UD |
62 | no longer enough. So the smallest entity will have to grow: @dfn{wide |
63 | characters} will now be used. Instead of one byte, two or four will | |
64 | be used instead. (Three are not good to address in memory and more | |
65 | than four bytes seem not to be necessary). | |
390955cb UD |
66 | |
67 | @cindex Unicode | |
68 | @cindex ISO 10646 | |
d731df03 | 69 | As shown in some other part of this manual, |
390955cb UD |
70 | @c !!! Ahem, wide char string functions are not yet covered -- drepper |
71 | there exists a completely new family of functions which can handle texts | |
aaca11d8 UD |
72 | of this kind in memory. The most commonly used character sets for such |
73 | internal wide character representations are Unicode and @w{ISO 10646} | |
74 | (also known as UCS for Universal Character Set). Unicode was originally | |
75 | planned as a 16-bit character set, whereas @w{ISO 10646} was designed to | |
76 | be a 31-bit large code space. The two standards are practically identical. | |
77 | They have the same character repertoire and code table, but Unicode specifies | |
78 | added semantics. At the moment, only characters in the first @code{0x10000} | |
79 | code positions (the so-called Basic Multilingual Plane, BMP) have been | |
80 | assigned, but the assignment of more specialized characters outside this | |
81 | 16-bit space is already in progress. A number of encodings have been | |
82 | defined for Unicode and @w{ISO 10646} characters: | |
83 | @cindex UCS-2 | |
84 | @cindex UCS-4 | |
85 | @cindex UTF-8 | |
86 | @cindex UTF-16 | |
87 | UCS-2 is a 16-bit word that can only represent characters | |
88 | from the BMP, UCS-4 is a 32-bit word than can represent any Unicode | |
89 | and @w{ISO 10646} character, UTF-8 is an ASCII compatible encoding where | |
90 | ASCII characters are represented by ASCII bytes and non-ASCII characters | |
91 | by sequences of 2-6 non-ASCII bytes, and finally UTF-16 is an extension | |
92 | of UCS-2 in which pairs of certain UCS-2 words can be used to encode | |
93 | non-BMP characters up to @code{0x10ffff}. | |
390955cb | 94 | |
d731df03 UD |
95 | To represent wide characters the @code{char} type is not suitable. For |
96 | this reason the @w{ISO C} standard introduces a new type which is | |
97 | designed to keep one character of a wide character string. To maintain | |
98 | the similarity there is also a type corresponding to @code{int} for | |
99 | those functions which take a single wide character. | |
390955cb UD |
100 | |
101 | @comment stddef.h | |
102 | @comment ISO | |
103 | @deftp {Data type} wchar_t | |
104 | This data type is used as the base type for wide character strings. | |
105 | I.e., arrays of objects of this type are the equivalent of @code{char[]} | |
106 | for multibyte character strings. The type is defined in @file{stddef.h}. | |
107 | ||
976780fd | 108 | The @w{ISO C90} standard, where this type was introduced, does not say |
390955cb | 109 | anything specific about the representation. It only requires that this |
aaca11d8 | 110 | type is capable of storing all elements of the basic character set. |
b160422e | 111 | Therefore it would be legitimate to define @code{wchar_t} as |
390955cb UD |
112 | @code{char}. This might make sense for embedded systems. |
113 | ||
114 | But for GNU systems this type is always 32 bits wide. It is therefore | |
aaca11d8 UD |
115 | capable of representing all UCS-4 values and therefore covering all of |
116 | @w{ISO 10646}. Some Unix systems define @code{wchar_t} as a 16-bit type and | |
390955cb | 117 | thereby follow Unicode very strictly. This is perfectly fine with the |
d731df03 | 118 | standard but it also means that to represent all characters from Unicode |
aaca11d8 UD |
119 | and @w{ISO 10646} one has to use UTF-16 surrogate characters which is in |
120 | fact a multi-wide-character encoding. But this contradicts the purpose | |
121 | of the @code{wchar_t} type. | |
390955cb UD |
122 | @end deftp |
123 | ||
124 | @comment wchar.h | |
125 | @comment ISO | |
126 | @deftp {Data type} wint_t | |
127 | @code{wint_t} is a data type used for parameters and variables which | |
128 | contain a single wide character. As the name already suggests it is the | |
129 | equivalent to @code{int} when using the normal @code{char} strings. The | |
130 | types @code{wchar_t} and @code{wint_t} have often the same | |
131 | representation if their size if 32 bits wide but if @code{wchar_t} is | |
132 | defined as @code{char} the type @code{wint_t} must be defined as | |
133 | @code{int} due to the parameter promotion. | |
134 | ||
135 | @pindex wchar.h | |
aaca11d8 UD |
136 | This type is defined in @file{wchar.h} and got introduced in |
137 | @w{Amendment 1} to @w{ISO C90}. | |
390955cb UD |
138 | @end deftp |
139 | ||
140 | As there are for the @code{char} data type there also exist macros | |
141 | specifying the minimum and maximum value representable in an object of | |
142 | type @code{wchar_t}. | |
143 | ||
144 | @comment wchar.h | |
145 | @comment ISO | |
146 | @deftypevr Macro wint_t WCHAR_MIN | |
147 | The macro @code{WCHAR_MIN} evaluates to the minimum value representable | |
148 | by an object of type @code{wint_t}. | |
149 | ||
aaca11d8 | 150 | This macro got introduced in @w{Amendment 1} to @w{ISO C90}. |
390955cb UD |
151 | @end deftypevr |
152 | ||
153 | @comment wchar.h | |
154 | @comment ISO | |
155 | @deftypevr Macro wint_t WCHAR_MAX | |
01f8c9f2 | 156 | The macro @code{WCHAR_MAX} evaluates to the maximum value representable |
390955cb UD |
157 | by an object of type @code{wint_t}. |
158 | ||
aaca11d8 | 159 | This macro got introduced in @w{Amendment 1} to @w{ISO C90}. |
390955cb UD |
160 | @end deftypevr |
161 | ||
162 | Another special wide character value is the equivalent to @code{EOF}. | |
163 | ||
164 | @comment wchar.h | |
165 | @comment ISO | |
166 | @deftypevr Macro wint_t WEOF | |
167 | The macro @code{WEOF} evaluates to a constant expression of type | |
168 | @code{wint_t} whose value is different from any member of the extended | |
169 | character set. | |
170 | ||
171 | @code{WEOF} need not be the same value as @code{EOF} and unlike | |
172 | @code{EOF} it also need @emph{not} be negative. I.e., sloppy code like | |
173 | ||
174 | @smallexample | |
175 | @{ | |
176 | int c; | |
177 | ... | |
178 | while ((c = getc (fp)) < 0) | |
179 | ... | |
180 | @} | |
181 | @end smallexample | |
182 | ||
183 | @noindent | |
184 | has to be rewritten to explicitly use @code{WEOF} when wide characters | |
185 | are used. | |
186 | ||
187 | @smallexample | |
188 | @{ | |
189 | wint_t c; | |
190 | ... | |
191 | while ((c = wgetc (fp)) != WEOF) | |
192 | ... | |
193 | @} | |
194 | @end smallexample | |
195 | ||
196 | @pindex wchar.h | |
aaca11d8 | 197 | This macro was introduced in @w{Amendment 1} to @w{ISO C90} and is |
390955cb UD |
198 | defined in @file{wchar.h}. |
199 | @end deftypevr | |
200 | ||
201 | ||
202 | These internal representations present problems when it comes to storing | |
d731df03 | 203 | and transmittal, since a single wide character consists of more |
390955cb UD |
204 | than one byte they are effected by byte-ordering. I.e., machines with |
205 | different endianesses would see different value accessing the same data. | |
206 | This also applies for communication protocols which are all byte-based | |
207 | and therefore the sender has to decide about splitting the wide | |
d731df03 | 208 | character in bytes. A last (but not least important) point is that wide |
390955cb UD |
209 | characters often require more storage space than an customized byte |
210 | oriented character set. | |
211 | ||
212 | @cindex multibyte character | |
d731df03 UD |
213 | @cindex EBCDIC |
214 | For all the above reasons, an external encoding which is different | |
aaca11d8 | 215 | from the internal encoding is often used if the latter is UCS-2 or UCS-4. |
d731df03 UD |
216 | The external encoding is byte-based and can be chosen appropriately for |
217 | the environment and for the texts to be handled. There exist a variety | |
218 | of different character sets which can be used for this external | |
219 | encoding. Information which will not be exhaustively presented | |
220 | here--instead, a description of the major groups will suffice. All of | |
221 | the ASCII-based character sets [_bkoz_: do you mean Roman character | |
222 | sets? If not, what do you mean here?] fulfill one requirement: they are | |
223 | "filesystem safe". This means that the character @code{'/'} is used in | |
224 | the encoding @emph{only} to represent itself. Things are a bit | |
225 | different for character sets like EBCDIC (Extended Binary Coded Decimal | |
226 | Interchange Code, a character set family used by IBM) but if the | |
390955cb UD |
227 | operation system does not understand EBCDIC directly the parameters to |
228 | system calls have to be converted first anyhow. | |
229 | ||
230 | @itemize @bullet | |
231 | @item | |
aaca11d8 | 232 | The simplest character sets are single-byte character sets. There can be |
390955cb UD |
233 | only up to 256 characters (for @w{8 bit} character sets) which is not |
234 | sufficient to cover all languages but might be sufficient to handle a | |
235 | specific text. Another reason to choose this is because of constraints | |
d731df03 | 236 | from interaction with other programs (which might not be 8-bit clean). |
390955cb UD |
237 | |
238 | @cindex ISO 2022 | |
239 | @item | |
240 | The @w{ISO 2022} standard defines a mechanism for extended character | |
241 | sets where one character @emph{can} be represented by more than one | |
242 | byte. This is achieved by associating a state with the text. Embedded | |
243 | in the text can be characters which can be used to change the state. | |
244 | Each byte in the text might have a different interpretation in each | |
245 | state. The state might even influence whether a given byte stands for a | |
246 | character on its own or whether it has to be combined with some more | |
247 | bytes. | |
248 | ||
249 | @cindex EUC | |
aa64639e | 250 | @cindex Shift_JIS |
390955cb UD |
251 | @cindex SJIS |
252 | In most uses of @w{ISO 2022} the defined character sets do not allow | |
253 | state changes which cover more than the next character. This has the | |
254 | big advantage that whenever one can identify the beginning of the byte | |
255 | sequence of a character one can interpret a text correctly. Examples of | |
256 | character sets using this policy are the various EUC character sets | |
257 | (used by Sun's operations systems, EUC-JP, EUC-KR, EUC-TW, and EUC-CN) | |
aa64639e | 258 | or Shift_JIS (SJIS, a Japanese encoding). |
390955cb UD |
259 | |
260 | But there are also character sets using a state which is valid for more | |
261 | than one character and has to be changed by another byte sequence. | |
262 | Examples for this are ISO-2022-JP, ISO-2022-KR, and ISO-2022-CN. | |
263 | ||
264 | @item | |
265 | @cindex ISO 6937 | |
266 | Early attempts to fix 8 bit character sets for other languages using the | |
267 | Roman alphabet lead to character sets like @w{ISO 6937}. Here bytes | |
d731df03 UD |
268 | representing characters like the acute accent do not produce output |
269 | themselves: one has to combine them with other characters to get the | |
270 | desired result. E.g., the byte sequence @code{0xc2 0x61} (non-spacing | |
271 | acute accent, following by lower-case `a') to get the ``small a with | |
640a3be1 | 272 | acute'' character. To get the acute accent character on its own, one has |
d731df03 | 273 | to write @code{0xc2 0x20} (the non-spacing acute followed by a space). |
390955cb | 274 | |
aaca11d8 UD |
275 | This type of character set is used in some embedded systems such as |
276 | teletex. | |
390955cb UD |
277 | |
278 | @item | |
279 | @cindex UTF-8 | |
aaca11d8 | 280 | Instead of converting the Unicode or @w{ISO 10646} text used internally, |
a58fe839 | 281 | it is often also sufficient to simply use an encoding different than |
aaca11d8 | 282 | UCS-2/UCS-4. The Unicode and @w{ISO 10646} standards even specify such an |
390955cb | 283 | encoding: UTF-8. This encoding is able to represent all of @w{ISO |
aaca11d8 | 284 | 10464} 31 bits in a byte string of length one to six. |
390955cb UD |
285 | |
286 | @cindex UTF-7 | |
287 | There were a few other attempts to encode @w{ISO 10646} such as UTF-7 | |
288 | but UTF-8 is today the only encoding which should be used. In fact, | |
aaca11d8 | 289 | UTF-8 will hopefully soon be the only external encoding that has to be |
d731df03 | 290 | supported. It proves to be universally usable and the only disadvantage |
aaca11d8 | 291 | is that it favors Roman languages by making the byte string |
390955cb | 292 | representation of other scripts (Cyrillic, Greek, Asian scripts) longer |
d731df03 UD |
293 | than necessary if using a specific character set for these scripts. |
294 | Methods like the Unicode compression scheme can alleviate these | |
295 | problems. | |
390955cb UD |
296 | @end itemize |
297 | ||
d731df03 UD |
298 | The question remaining is: how to select the character set or encoding |
299 | to use. The answer: you cannot decide about it yourself, it is decided | |
300 | by the developers of the system or the majority of the users. Since the | |
301 | goal is interoperability one has to use whatever the other people one | |
302 | works with use. If there are no constraints the selection is based on | |
303 | the requirements the expected circle of users will have. I.e., if a | |
304 | project is expected to only be used in, say, Russia it is fine to use | |
305 | KOI8-R or a similar character set. But if at the same time people from, | |
a58fe839 | 306 | say, Greece are participating one should use a character set which allows |
d731df03 UD |
307 | all people to collaborate. |
308 | ||
309 | The most widely useful solution seems to be: go with the most general | |
310 | character set, namely @w{ISO 10646}. Use UTF-8 as the external encoding | |
311 | and problems about users not being able to use their own language | |
312 | adequately are a thing of the past. | |
390955cb UD |
313 | |
314 | One final comment about the choice of the wide character representation | |
315 | is necessary at this point. We have said above that the natural choice | |
d01fe032 UD |
316 | is using Unicode or @w{ISO 10646}. This is not required, but at least |
317 | encouraged, by the @w{ISO C} standard. The standard defines at least a | |
318 | macro @code{__STDC_ISO_10646__} that is only defined on systems where | |
319 | the @code{wchar_t} type encodes @w{ISO 10646} characters. If this | |
320 | symbol is not defined one should as much as possible avoid making | |
321 | assumption about the wide character representation. If the programmer | |
322 | uses only the functions provided by the C library to handle wide | |
323 | character strings there should not be any compatibility problems with | |
324 | other systems. | |
390955cb UD |
325 | |
326 | @node Charset Function Overview | |
327 | @section Overview about Character Handling Functions | |
328 | ||
329 | A Unix @w{C library} contains three different sets of functions in two | |
7be8096f | 330 | families to handle character set conversion. The one function family |
390955cb UD |
331 | is specified in the @w{ISO C} standard and therefore is portable even |
332 | beyond the Unix world. | |
333 | ||
976780fd | 334 | The most commonly known set of functions, coming from the @w{ISO C90} |
390955cb UD |
335 | standard, is unfortunately the least useful one. In fact, these |
336 | functions should be avoided whenever possible, especially when | |
337 | developing libraries (as opposed to applications). | |
338 | ||
d731df03 | 339 | The second family of functions got introduced in the early Unix standards |
390955cb UD |
340 | (XPG2) and is still part of the latest and greatest Unix standard: |
341 | @w{Unix 98}. It is also the most powerful and useful set of functions. | |
aaca11d8 | 342 | But we will start with the functions defined in @w{Amendment 1} to |
976780fd | 343 | @w{ISO C90}. |
390955cb UD |
344 | |
345 | @node Restartable multibyte conversion | |
346 | @section Restartable Multibyte Conversion Functions | |
347 | ||
348 | The @w{ISO C} standard defines functions to convert strings from a | |
349 | multibyte representation to wide character strings. There are a number | |
350 | of peculiarities: | |
351 | ||
352 | @itemize @bullet | |
353 | @item | |
354 | The character set assumed for the multibyte encoding is not specified | |
355 | as an argument to the functions. Instead the character set specified by | |
356 | the @code{LC_CTYPE} category of the current locale is used; see | |
357 | @ref{Locale Categories}. | |
358 | ||
359 | @item | |
360 | The functions handling more than one character at a time require NUL | |
361 | terminated strings as the argument. I.e., converting blocks of text | |
362 | does not work unless one can add a NUL byte at an appropriate place. | |
363 | The GNU C library contains some extensions the standard which allow | |
364 | specifying a size but basically they also expect terminated strings. | |
365 | @end itemize | |
366 | ||
367 | Despite these limitations the @w{ISO C} functions can very well be used | |
368 | in many contexts. In graphical user interfaces, for instance, it is not | |
369 | uncommon to have functions which require text to be displayed in a wide | |
370 | character string if it is not simple ASCII. The text itself might come | |
7be8096f UD |
371 | from a file with translations and the user should decide about the |
372 | current locale which determines the translation and therefore also the | |
373 | external encoding used. In such a situation (and many others) the | |
390955cb UD |
374 | functions described here are perfect. If more freedom while performing |
375 | the conversion is necessary take a look at the @code{iconv} functions | |
464d646f | 376 | (@pxref{Generic Charset Conversion}). |
390955cb UD |
377 | |
378 | @menu | |
379 | * Selecting the Conversion:: Selecting the conversion and its properties. | |
380 | * Keeping the state:: Representing the state of the conversion. | |
381 | * Converting a Character:: Converting Single Characters. | |
382 | * Converting Strings:: Converting Multibyte and Wide Character | |
383 | Strings. | |
384 | * Multibyte Conversion Example:: A Complete Multibyte Conversion Example. | |
385 | @end menu | |
386 | ||
387 | @node Selecting the Conversion | |
388 | @subsection Selecting the conversion and its properties | |
389 | ||
390 | We already said above that the currently selected locale for the | |
391 | @code{LC_CTYPE} category decides about the conversion which is performed | |
392 | by the functions we are about to describe. Each locale uses its own | |
393 | character set (given as an argument to @code{localedef}) and this is the | |
394 | one assumed as the external multibyte encoding. The wide character | |
aaca11d8 | 395 | character set always is UCS-4, at least on GNU systems. |
390955cb UD |
396 | |
397 | A characteristic of each multibyte character set is the maximum number | |
398 | of bytes which can be necessary to represent one character. This | |
399 | information is quite important when writing code which uses the | |
400 | conversion functions. In the examples below we will see some examples. | |
401 | The @w{ISO C} standard defines two macros which provide this information. | |
402 | ||
403 | ||
404 | @comment limits.h | |
405 | @comment ISO | |
406 | @deftypevr Macro int MB_LEN_MAX | |
407 | This macro specifies the maximum number of bytes in the multibyte | |
408 | sequence for a single character in any of the supported locales. It is | |
409 | a compile-time constant and it is defined in @file{limits.h}. | |
410 | @pindex limits.h | |
411 | @end deftypevr | |
412 | ||
413 | @comment stdlib.h | |
414 | @comment ISO | |
415 | @deftypevr Macro int MB_CUR_MAX | |
416 | @code{MB_CUR_MAX} expands into a positive integer expression that is the | |
417 | maximum number of bytes in a multibyte character in the current locale. | |
418 | The value is never greater than @code{MB_LEN_MAX}. Unlike | |
419 | @code{MB_LEN_MAX} this macro need not be a compile-time constant and in | |
420 | fact, in the GNU C library it is not. | |
421 | ||
422 | @pindex stdlib.h | |
423 | @code{MB_CUR_MAX} is defined in @file{stdlib.h}. | |
424 | @end deftypevr | |
425 | ||
976780fd | 426 | Two different macros are necessary since strictly @w{ISO C90} compilers |
390955cb UD |
427 | do not allow variable length array definitions but still it is desirable |
428 | to avoid dynamic allocation. This incomplete piece of code shows the | |
429 | problem: | |
430 | ||
431 | @smallexample | |
432 | @{ | |
433 | char buf[MB_LEN_MAX]; | |
434 | ssize_t len = 0; | |
435 | ||
436 | while (! feof (fp)) | |
437 | @{ | |
438 | fread (&buf[len], 1, MB_CUR_MAX - len, fp); | |
439 | /* @r{... process} buf */ | |
440 | len -= used; | |
441 | @} | |
442 | @} | |
443 | @end smallexample | |
444 | ||
445 | The code in the inner loop is expected to have always enough bytes in | |
446 | the array @var{buf} to convert one multibyte character. The array | |
447 | @var{buf} has to be sized statically since many compilers do not allow a | |
448 | variable size. The @code{fread} call makes sure that always | |
d731df03 UD |
449 | @code{MB_CUR_MAX} bytes are available in @var{buf}. Note that it isn't |
450 | a problem if @code{MB_CUR_MAX} is not a compile-time constant. | |
390955cb UD |
451 | |
452 | ||
453 | @node Keeping the state | |
454 | @subsection Representing the state of the conversion | |
455 | ||
456 | @cindex stateful | |
457 | In the introduction of this chapter it was said that certain character | |
458 | sets use a @dfn{stateful} encoding. I.e., the encoded values depend in | |
7be8096f | 459 | some way on the previous bytes in the text. |
390955cb UD |
460 | |
461 | Since the conversion functions allow converting a text in more than one | |
462 | step we must have a way to pass this information from one call of the | |
463 | functions to another. | |
464 | ||
465 | @comment wchar.h | |
466 | @comment ISO | |
467 | @deftp {Data type} mbstate_t | |
468 | @cindex shift state | |
469 | A variable of type @code{mbstate_t} can contain all the information | |
470 | about the @dfn{shift state} needed from one call to a conversion | |
471 | function to another. | |
472 | ||
473 | @pindex wchar.h | |
aaca11d8 UD |
474 | This type is defined in @file{wchar.h}. It got introduced in |
475 | @w{Amendment 1} to @w{ISO C90}. | |
390955cb UD |
476 | @end deftp |
477 | ||
478 | To use objects of this type the programmer has to define such objects | |
479 | (normally as local variables on the stack) and pass a pointer to the | |
480 | object to the conversion functions. This way the conversion function | |
481 | can update the object if the current multibyte character set is | |
482 | stateful. | |
483 | ||
484 | There is no specific function or initializer to put the state object in | |
485 | any specific state. The rules are that the object should always | |
486 | represent the initial state before the first use and this is achieved by | |
487 | clearing the whole variable with code such as follows: | |
488 | ||
489 | @smallexample | |
490 | @{ | |
491 | mbstate_t state; | |
492 | memset (&state, '\0', sizeof (state)); | |
493 | /* @r{from now on @var{state} can be used.} */ | |
494 | ... | |
495 | @} | |
496 | @end smallexample | |
497 | ||
498 | When using the conversion functions to generate output it is often | |
7be8096f | 499 | necessary to test whether the current state corresponds to the initial |
390955cb UD |
500 | state. This is necessary, for example, to decide whether or not to emit |
501 | escape sequences to set the state to the initial state at certain | |
502 | sequence points. Communication protocols often require this. | |
503 | ||
504 | @comment wchar.h | |
505 | @comment ISO | |
506 | @deftypefun int mbsinit (const mbstate_t *@var{ps}) | |
507 | This function determines whether the state object pointed to by @var{ps} | |
7be8096f | 508 | is in the initial state or not. If @var{ps} is a null pointer or the |
390955cb UD |
509 | object is in the initial state the return value is nonzero. Otherwise |
510 | it is zero. | |
511 | ||
512 | @pindex wchar.h | |
aaca11d8 | 513 | This function was introduced in @w{Amendment 1} to @w{ISO C90} and |
390955cb UD |
514 | is declared in @file{wchar.h}. |
515 | @end deftypefun | |
516 | ||
517 | Code using this function often looks similar to this: | |
518 | ||
37a87f83 UD |
519 | @c Fix the example to explicitly say how to generate the escape sequence |
520 | @c to restore the initial state. | |
390955cb UD |
521 | @smallexample |
522 | @{ | |
523 | mbstate_t state; | |
524 | memset (&state, '\0', sizeof (state)); | |
525 | /* @r{Use @var{state}.} */ | |
526 | ... | |
527 | if (! mbsinit (&state)) | |
528 | @{ | |
529 | /* @r{Emit code to return to initial state.} */ | |
976780fd UD |
530 | const wchar_t empty[] = L""; |
531 | const wchar_t *srcp = empty; | |
37a87f83 | 532 | wcsrtombs (outbuf, &srcp, outbuflen, &state); |
390955cb UD |
533 | @} |
534 | ... | |
535 | @} | |
536 | @end smallexample | |
537 | ||
37a87f83 UD |
538 | The code to emit the escape sequence to get back to the initial state is |
539 | interesting. The @code{wcsrtombs} function can be used to determine the | |
540 | necessary output code (@pxref{Converting Strings}). Please note that on | |
541 | GNU systems it is not necessary to perform this extra action for the | |
49c091e5 | 542 | conversion from multibyte text to wide character text since the wide |
37a87f83 UD |
543 | character encoding is not stateful. But there is nothing mentioned in |
544 | any standard which prohibits making @code{wchar_t} using a stateful | |
545 | encoding. | |
546 | ||
390955cb UD |
547 | @node Converting a Character |
548 | @subsection Converting Single Characters | |
549 | ||
550 | The most fundamental of the conversion functions are those dealing with | |
551 | single characters. Please note that this does not always mean single | |
552 | bytes. But since there is very often a subset of the multibyte | |
553 | character set which consists of single byte sequences there are | |
554 | functions to help with converting bytes. One very important and often | |
555 | applicable scenario is where ASCII is a subpart of the multibyte | |
556 | character set. I.e., all ASCII characters stand for itself and all | |
557 | other characters have at least a first byte which is beyond the range | |
558 | @math{0} to @math{127}. | |
559 | ||
560 | @comment wchar.h | |
561 | @comment ISO | |
562 | @deftypefun wint_t btowc (int @var{c}) | |
563 | The @code{btowc} function (``byte to wide character'') converts a valid | |
7be8096f UD |
564 | single byte character @var{c} in the initial shift state into the wide |
565 | character equivalent using the conversion rules from the currently | |
566 | selected locale of the @code{LC_CTYPE} category. | |
390955cb UD |
567 | |
568 | If @code{(unsigned char) @var{c}} is no valid single byte multibyte | |
569 | character or if @var{c} is @code{EOF} the function returns @code{WEOF}. | |
570 | ||
571 | Please note the restriction of @var{c} being tested for validity only in | |
572 | the initial shift state. There is no @code{mbstate_t} object used from | |
573 | which the state information is taken and the function also does not use | |
574 | any static state. | |
575 | ||
576 | @pindex wchar.h | |
aaca11d8 | 577 | This function was introduced in @w{Amendment 1} to @w{ISO C90} and |
390955cb UD |
578 | is declared in @file{wchar.h}. |
579 | @end deftypefun | |
580 | ||
581 | Despite the limitation that the single byte value always is interpreted | |
582 | in the initial state this function is actually useful most of the time. | |
d731df03 | 583 | Most characters are either entirely single-byte character sets or they |
390955cb | 584 | are extension to ASCII. But then it is possible to write code like this |
7be8096f | 585 | (not that this specific example is very useful): |
390955cb UD |
586 | |
587 | @smallexample | |
588 | wchar_t * | |
589 | itow (unsigned long int val) | |
590 | @{ | |
591 | static wchar_t buf[30]; | |
592 | wchar_t *wcp = &buf[29]; | |
593 | *wcp = L'\0'; | |
594 | while (val != 0) | |
595 | @{ | |
596 | *--wcp = btowc ('0' + val % 10); | |
597 | val /= 10; | |
598 | @} | |
599 | if (wcp == &buf[29]) | |
d731df03 | 600 | *--wcp = L'0'; |
390955cb UD |
601 | return wcp; |
602 | @} | |
603 | @end smallexample | |
604 | ||
d731df03 | 605 | Why is it necessary to use such a complicated implementation and not |
7be8096f | 606 | simply cast @code{'0' + val % 10} to a wide character? The answer is |
d731df03 UD |
607 | that there is no guarantee that one can perform this kind of arithmetic |
608 | on the character of the character set used for @code{wchar_t} | |
7be8096f UD |
609 | representation. In other situations the bytes are not constant at |
610 | compile time and so the compiler cannot do the work. In situations like | |
611 | this it is necessary @code{btowc}. | |
390955cb | 612 | |
d731df03 | 613 | @noindent |
390955cb UD |
614 | There also is a function for the conversion in the other direction. |
615 | ||
616 | @comment wchar.h | |
617 | @comment ISO | |
618 | @deftypefun int wctob (wint_t @var{c}) | |
619 | The @code{wctob} function (``wide character to byte'') takes as the | |
c1b2d472 | 620 | parameter a valid wide character. If the multibyte representation for |
390955cb UD |
621 | this character in the initial state is exactly one byte long the return |
622 | value of this function is this character. Otherwise the return value is | |
623 | @code{EOF}. | |
624 | ||
625 | @pindex wchar.h | |
aaca11d8 | 626 | This function was introduced in @w{Amendment 1} to @w{ISO C90} and |
390955cb UD |
627 | is declared in @file{wchar.h}. |
628 | @end deftypefun | |
629 | ||
630 | There are more general functions to convert single character from | |
631 | multibyte representation to wide characters and vice versa. These | |
632 | functions pose no limit on the length of the multibyte representation | |
633 | and they also do not require it to be in the initial state. | |
634 | ||
635 | @comment wchar.h | |
636 | @comment ISO | |
637 | @deftypefun size_t mbrtowc (wchar_t *restrict @var{pwc}, const char *restrict @var{s}, size_t @var{n}, mbstate_t *restrict @var{ps}) | |
638 | @cindex stateful | |
639 | The @code{mbrtowc} function (``multibyte restartable to wide | |
640 | character'') converts the next multibyte character in the string pointed | |
641 | to by @var{s} into a wide character and stores it in the wide character | |
642 | string pointed to by @var{pwc}. The conversion is performed according | |
643 | to the locale currently selected for the @code{LC_CTYPE} category. If | |
7be8096f UD |
644 | the conversion for the character set used in the locale requires a state |
645 | the multibyte string is interpreted in the state represented by the | |
bafb8ee9 UD |
646 | object pointed to by @var{ps}. If @var{ps} is a null pointer, a static, |
647 | internal state variable used only by the @code{mbrtowc} function is | |
7be8096f | 648 | used. |
390955cb UD |
649 | |
650 | If the next multibyte character corresponds to the NUL wide character | |
651 | the return value of the function is @math{0} and the state object is | |
652 | afterwards in the initial state. If the next @var{n} or fewer bytes | |
653 | form a correct multibyte character the return value is the number of | |
654 | bytes starting from @var{s} which form the multibyte character. The | |
655 | conversion state is updated according to the bytes consumed in the | |
656 | conversion. In both cases the wide character (either the @code{L'\0'} | |
657 | or the one found in the conversion) is stored in the string pointer to | |
658 | by @var{pwc} iff @var{pwc} is not null. | |
659 | ||
660 | If the first @var{n} bytes of the multibyte string possibly form a valid | |
661 | multibyte character but there are more than @var{n} bytes needed to | |
662 | complete it the return value of the function is @code{(size_t) -2} and | |
663 | no value is stored. Please note that this can happen even if @var{n} | |
664 | has a value greater or equal to @code{MB_CUR_MAX} since the input might | |
665 | contain redundant shift sequences. | |
666 | ||
7be8096f UD |
667 | If the first @code{n} bytes of the multibyte string cannot possibly form |
668 | a valid multibyte character also no value is stored, the global variable | |
669 | @code{errno} is set to the value @code{EILSEQ} and the function returns | |
390955cb UD |
670 | @code{(size_t) -1}. The conversion state is afterwards undefined. |
671 | ||
672 | @pindex wchar.h | |
aaca11d8 | 673 | This function was introduced in @w{Amendment 1} to @w{ISO C90} and |
390955cb UD |
674 | is declared in @file{wchar.h}. |
675 | @end deftypefun | |
676 | ||
677 | Using this function is straight forward. A function which copies a | |
678 | multibyte string into a wide character string while at the same time | |
679 | converting all lowercase character into uppercase could look like this | |
680 | (this is not the final version, just an example; it has no error | |
7be8096f | 681 | checking, and leaks sometimes memory): |
390955cb UD |
682 | |
683 | @smallexample | |
684 | wchar_t * | |
685 | mbstouwcs (const char *s) | |
686 | @{ | |
687 | size_t len = strlen (s); | |
688 | wchar_t *result = malloc ((len + 1) * sizeof (wchar_t)); | |
689 | wchar_t *wcp = result; | |
690 | wchar_t tmp[1]; | |
691 | mbstate_t state; | |
390955cb | 692 | size_t nbytes; |
b71f84a0 AJ |
693 | |
694 | memset (&state, '\0', sizeof (state)); | |
390955cb UD |
695 | while ((nbytes = mbrtowc (tmp, s, len, &state)) > 0) |
696 | @{ | |
697 | if (nbytes >= (size_t) -2) | |
698 | /* Invalid input string. */ | |
699 | return NULL; | |
700 | *result++ = towupper (tmp[0]); | |
701 | len -= nbytes; | |
702 | s += nbytes; | |
703 | @} | |
704 | return result; | |
705 | @} | |
706 | @end smallexample | |
707 | ||
708 | The use of @code{mbrtowc} should be clear. A single wide character is | |
709 | stored in @code{@var{tmp}[0]} and the number of consumed bytes is stored | |
710 | in the variable @var{nbytes}. In case the the conversion was successful | |
711 | the uppercase variant of the wide character is stored in the | |
712 | @var{result} array and the pointer to the input string and the number of | |
713 | available bytes is adjusted. | |
714 | ||
715 | The only non-obvious thing about the function might be the way memory is | |
716 | allocated for the result. The above code uses the fact that there can | |
717 | never be more wide characters in the converted results than there are | |
718 | bytes in the multibyte input string. This method yields to a | |
719 | pessimistic guess about the size of the result and if many wide | |
720 | character strings have to be constructed this way or the strings are | |
7be8096f | 721 | long, the extra memory required allocated because the input string |
49c091e5 | 722 | contains multibyte characters might be significant. It would be |
7be8096f UD |
723 | possible to resize the allocated memory block to the correct size before |
724 | returning it. A better solution might be to allocate just the right | |
725 | amount of space for the result right away. Unfortunately there is no | |
726 | function to compute the length of the wide character string directly | |
727 | from the multibyte string. But there is a function which does part of | |
728 | the work. | |
390955cb UD |
729 | |
730 | @comment wchar.h | |
731 | @comment ISO | |
732 | @deftypefun size_t mbrlen (const char *restrict @var{s}, size_t @var{n}, mbstate_t *@var{ps}) | |
733 | The @code{mbrlen} function (``multibyte restartable length'') computes | |
734 | the number of at most @var{n} bytes starting at @var{s} which form the | |
735 | next valid and complete multibyte character. | |
736 | ||
737 | If the next multibyte character corresponds to the NUL wide character | |
738 | the return value is @math{0}. If the next @var{n} bytes form a valid | |
739 | multibyte character the number of bytes belonging to this multibyte | |
740 | character byte sequence is returned. | |
741 | ||
742 | If the the first @var{n} bytes possibly form a valid multibyte | |
743 | character but it is incomplete the return value is @code{(size_t) -2}. | |
744 | Otherwise the multibyte character sequence is invalid and the return | |
745 | value is @code{(size_t) -1}. | |
746 | ||
747 | The multibyte sequence is interpreted in the state represented by the | |
bafb8ee9 | 748 | object pointed to by @var{ps}. If @var{ps} is a null pointer, a state |
390955cb UD |
749 | object local to @code{mbrlen} is used. |
750 | ||
751 | @pindex wchar.h | |
aaca11d8 | 752 | This function was introduced in @w{Amendment 1} to @w{ISO C90} and |
390955cb UD |
753 | is declared in @file{wchar.h}. |
754 | @end deftypefun | |
755 | ||
756 | The tentative reader now will of course note that @code{mbrlen} can be | |
757 | implemented as | |
758 | ||
759 | @smallexample | |
760 | mbrtowc (NULL, s, n, ps != NULL ? ps : &internal) | |
761 | @end smallexample | |
762 | ||
763 | This is true and in fact is mentioned in the official specification. | |
764 | Now, how can this function be used to determine the length of the wide | |
765 | character string created from a multibyte character string? It is not | |
766 | directly usable but we can define a function @code{mbslen} using it: | |
767 | ||
768 | @smallexample | |
769 | size_t | |
770 | mbslen (const char *s) | |
771 | @{ | |
772 | mbstate_t state; | |
773 | size_t result = 0; | |
774 | size_t nbytes; | |
775 | memset (&state, '\0', sizeof (state)); | |
776 | while ((nbytes = mbrlen (s, MB_LEN_MAX, &state)) > 0) | |
777 | @{ | |
778 | if (nbytes >= (size_t) -2) | |
779 | /* @r{Something is wrong.} */ | |
780 | return (size_t) -1; | |
781 | s += nbytes; | |
782 | ++result; | |
783 | @} | |
784 | return result; | |
785 | @} | |
786 | @end smallexample | |
787 | ||
788 | This function simply calls @code{mbrlen} for each multibyte character | |
789 | in the string and counts the number of function calls. Please note that | |
790 | we here use @code{MB_LEN_MAX} as the size argument in the @code{mbrlen} | |
791 | call. This is OK since a) this value is larger then the length of the | |
792 | longest multibyte character sequence and b) because we know that the | |
7be8096f UD |
793 | string @var{s} ends with a NUL byte which cannot be part of any other |
794 | multibyte character sequence but the one representing the NUL wide | |
390955cb UD |
795 | character. Therefore the @code{mbrlen} function will never read invalid |
796 | memory. | |
797 | ||
798 | Now that this function is available (just to make this clear, this | |
799 | function is @emph{not} part of the GNU C library) we can compute the | |
800 | number of wide character required to store the converted multibyte | |
801 | character string @var{s} using | |
802 | ||
803 | @smallexample | |
804 | wcs_bytes = (mbslen (s) + 1) * sizeof (wchar_t); | |
805 | @end smallexample | |
806 | ||
807 | Please note that the @code{mbslen} function is quite inefficient. The | |
808 | implementation of @code{mbstouwcs} implemented using @code{mbslen} would | |
809 | have to perform the conversion of the multibyte character input string | |
810 | twice and this conversion might be quite expensive. So it is necessary | |
c1b2d472 | 811 | to think about the consequences of using the easier but imprecise method |
390955cb UD |
812 | before doing the work twice. |
813 | ||
814 | @comment wchar.h | |
815 | @comment ISO | |
816 | @deftypefun size_t wcrtomb (char *restrict @var{s}, wchar_t @var{wc}, mbstate_t *restrict @var{ps}) | |
817 | The @code{wcrtomb} function (``wide character restartable to | |
818 | multibyte'') converts a single wide character into a multibyte string | |
819 | corresponding to that wide character. | |
820 | ||
7be8096f UD |
821 | If @var{s} is a null pointer the function resets the the state stored in |
822 | the objects pointer to by @var{ps} (or the internal @code{mbstate_t} | |
823 | object) to the initial state. This can also be achieved by a call like | |
824 | this: | |
390955cb UD |
825 | |
826 | @smallexample | |
827 | wcrtombs (temp_buf, L'\0', ps) | |
828 | @end smallexample | |
829 | ||
830 | @noindent | |
7be8096f | 831 | since if @var{s} is a null pointer @code{wcrtomb} performs as if it |
390955cb UD |
832 | writes into an internal buffer which is guaranteed to be large enough. |
833 | ||
834 | If @var{wc} is the NUL wide character @code{wcrtomb} emits, if | |
835 | necessary, a shift sequence to get the state @var{ps} into the initial | |
836 | state followed by a single NUL byte is stored in the string @var{s}. | |
837 | ||
838 | Otherwise a byte sequence (possibly including shift sequences) is | |
7be8096f UD |
839 | written into the string @var{s}. This of only happens if @var{wc} is a |
840 | valid wide character, i.e., it has a multibyte representation in the | |
841 | character set selected by locale of the @code{LC_CTYPE} category. If | |
842 | @var{wc} is no valid wide character nothing is stored in the strings | |
843 | @var{s}, @code{errno} is set to @code{EILSEQ}, the conversion state in | |
844 | @var{ps} is undefined and the return value is @code{(size_t) -1}. | |
390955cb UD |
845 | |
846 | If no error occurred the function returns the number of bytes stored in | |
847 | the string @var{s}. This includes all byte representing shift | |
848 | sequences. | |
849 | ||
850 | One word about the interface of the function: there is no parameter | |
851 | specifying the length of the array @var{s}. Instead the function | |
852 | assumes that there are at least @code{MB_CUR_MAX} bytes available since | |
853 | this is the maximum length of any byte sequence representing a single | |
854 | character. So the caller has to make sure that there is enough space | |
855 | available, otherwise buffer overruns can occur. | |
856 | ||
857 | @pindex wchar.h | |
aaca11d8 | 858 | This function was introduced in @w{Amendment 1} to @w{ISO C90} and is |
390955cb UD |
859 | declared in @file{wchar.h}. |
860 | @end deftypefun | |
861 | ||
862 | Using this function is as easy as using @code{mbrtowc}. The following | |
863 | example appends a wide character string to a multibyte character string. | |
7be8096f UD |
864 | Again, the code is not really useful (and correct), it is simply here to |
865 | demonstrate the use and some problems. | |
390955cb UD |
866 | |
867 | @smallexample | |
868 | char * | |
976780fd | 869 | mbscatwcs (char *s, size_t len, const wchar_t *ws) |
390955cb UD |
870 | @{ |
871 | mbstate_t state; | |
7be8096f | 872 | /* @r{Find the end of the existing string.} */ |
390955cb UD |
873 | char *wp = strchr (s, '\0'); |
874 | len -= wp - s; | |
875 | memset (&state, '\0', sizeof (state)); | |
876 | do | |
877 | @{ | |
878 | size_t nbytes; | |
879 | if (len < MB_CUR_LEN) | |
880 | @{ | |
881 | /* @r{We cannot guarantee that the next} | |
882 | @r{character fits into the buffer, so} | |
883 | @r{return an error.} */ | |
884 | errno = E2BIG; | |
885 | return NULL; | |
886 | @} | |
887 | nbytes = wcrtomb (wp, *ws, &state); | |
888 | if (nbytes == (size_t) -1) | |
889 | /* @r{Error in the conversion.} */ | |
890 | return NULL; | |
891 | len -= nbytes; | |
892 | wp += nbytes; | |
893 | @} | |
894 | while (*ws++ != L'\0'); | |
895 | return s; | |
896 | @} | |
897 | @end smallexample | |
898 | ||
899 | First the function has to find the end of the string currently in the | |
900 | array @var{s}. The @code{strchr} call does this very efficiently since a | |
901 | requirement for multibyte character representations is that the NUL byte | |
902 | never is used except to represent itself (and in this context, the end | |
903 | of the string). | |
904 | ||
905 | After initializing the state object the loop is entered where the first | |
906 | task is to make sure there is enough room in the array @var{s}. We | |
907 | abort if there are not at least @code{MB_CUR_LEN} bytes available. This | |
908 | is not always optimal but we have no other choice. We might have less | |
909 | than @code{MB_CUR_LEN} bytes available but the next multibyte character | |
910 | might also be only one byte long. At the time the @code{wcrtomb} call | |
911 | returns it is too late to decide whether the buffer was large enough or | |
912 | not. If this solution is really unsuitable there is a very slow but | |
913 | more accurate solution. | |
914 | ||
915 | @smallexample | |
916 | ... | |
917 | if (len < MB_CUR_LEN) | |
918 | @{ | |
919 | mbstate_t temp_state; | |
920 | memcpy (&temp_state, &state, sizeof (state)); | |
921 | if (wcrtomb (NULL, *ws, &temp_state) > len) | |
922 | @{ | |
923 | /* @r{We cannot guarantee that the next} | |
924 | @r{character fits into the buffer, so} | |
925 | @r{return an error.} */ | |
926 | errno = E2BIG; | |
927 | return NULL; | |
928 | @} | |
929 | @} | |
930 | ... | |
931 | @end smallexample | |
932 | ||
933 | Here we do perform the conversion which might overflow the buffer so | |
934 | that we are afterwards in the position to make an exact decision about | |
935 | the buffer size. Please note the @code{NULL} argument for the | |
936 | destination buffer in the new @code{wcrtomb} call; since we are not | |
7be8096f UD |
937 | interested in the converted text at this point this is a nice way to |
938 | express this. The most unusual thing about this piece of code certainly | |
939 | is the duplication of the conversion state object. But think about | |
940 | this: if a change of the state is necessary to emit the next multibyte | |
941 | character we want to have the same shift state change performed in the | |
942 | real conversion. Therefore we have to preserve the initial shift state | |
390955cb UD |
943 | information. |
944 | ||
945 | There are certainly many more and even better solutions to this problem. | |
946 | This example is only meant for educational purposes. | |
947 | ||
948 | @node Converting Strings | |
949 | @subsection Converting Multibyte and Wide Character Strings | |
950 | ||
951 | The functions described in the previous section only convert a single | |
952 | character at a time. Most operations to be performed in real-world | |
953 | programs include strings and therefore the @w{ISO C} standard also | |
d731df03 UD |
954 | defines conversions on entire strings. However, the defined set of |
955 | functions is quite limited, thus the GNU C library contains a few | |
7be8096f | 956 | extensions which can help in some important situations. |
390955cb UD |
957 | |
958 | @comment wchar.h | |
959 | @comment ISO | |
960 | @deftypefun size_t mbsrtowcs (wchar_t *restrict @var{dst}, const char **restrict @var{src}, size_t @var{len}, mbstate_t *restrict @var{ps}) | |
961 | The @code{mbsrtowcs} function (``multibyte string restartable to wide | |
962 | character string'') converts an NUL terminated multibyte character | |
963 | string at @code{*@var{src}} into an equivalent wide character string, | |
964 | including the NUL wide character at the end. The conversion is started | |
965 | using the state information from the object pointed to by @var{ps} or | |
966 | from an internal object of @code{mbsrtowcs} if @var{ps} is a null | |
967 | pointer. Before returning the state object to match the state after the | |
968 | last converted character. The state is the initial state if the | |
969 | terminating NUL byte is reached and converted. | |
970 | ||
971 | If @var{dst} is not a null pointer the result is stored in the array | |
972 | pointed to by @var{dst}, otherwise the conversion result is not | |
973 | available since it is stored in an internal buffer. | |
974 | ||
975 | If @var{len} wide characters are stored in the array @var{dst} before | |
976 | reaching the end of the input string the conversion stops and @var{len} | |
977 | is returned. If @var{dst} is a null pointer @var{len} is never checked. | |
978 | ||
979 | Another reason for a premature return from the function call is if the | |
980 | input string contains an invalid multibyte sequence. In this case the | |
981 | global variable @code{errno} is set to @code{EILSEQ} and the function | |
982 | returns @code{(size_t) -1}. | |
983 | ||
984 | @c XXX The ISO C9x draft seems to have a problem here. It says that PS | |
985 | @c is not updated if DST is NULL. This is not said straight forward and | |
986 | @c none of the other functions is described like this. It would make sense | |
987 | @c to define the function this way but I don't think it is meant like this. | |
988 | ||
989 | In all other cases the function returns the number of wide characters | |
990 | converted during this call. If @var{dst} is not null @code{mbsrtowcs} | |
991 | stores in the pointer pointed to by @var{src} a null pointer (if the NUL | |
992 | byte in the input string was reached) or the address of the byte | |
993 | following the last converted multibyte character. | |
994 | ||
995 | @pindex wchar.h | |
aaca11d8 | 996 | This function was introduced in @w{Amendment 1} to @w{ISO C90} and is |
390955cb UD |
997 | declared in @file{wchar.h}. |
998 | @end deftypefun | |
999 | ||
1000 | The definition of this function has one limitation which has to be | |
1001 | understood. The requirement that @var{dst} has to be a NUL terminated | |
1002 | string provides problems if one wants to convert buffers with text. A | |
1003 | buffer is normally no collection of NUL terminated strings but instead a | |
1004 | continuous collection of lines, separated by newline characters. Now | |
1005 | assume a function to convert one line from a buffer is needed. Since | |
1006 | the line is not NUL terminated the source pointer cannot directly point | |
1007 | into the unmodified text buffer. This means, either one inserts the NUL | |
1008 | byte at the appropriate place for the time of the @code{mbsrtowcs} | |
1009 | function call (which is not doable for a read-only buffer or in a | |
1010 | multi-threaded application) or one copies the line in an extra buffer | |
1011 | where it can be terminated by a NUL byte. Note that it is not in | |
1012 | general possible to limit the number of characters to convert by setting | |
1013 | the parameter @var{len} to any specific value. Since it is not known | |
1014 | how many bytes each multibyte character sequence is in length one always | |
1015 | could do only a guess. | |
1016 | ||
1017 | @cindex stateful | |
1018 | There is still a problem with the method of NUL-terminating a line right | |
1019 | after the newline character which could lead to very strange results. | |
1020 | As said in the description of the @var{mbsrtowcs} function above the | |
1021 | conversion state is guaranteed to be in the initial shift state after | |
1022 | processing the NUL byte at the end of the input string. But this NUL | |
1023 | byte is not really part of the text. I.e., the conversion state after | |
1024 | the newline in the original text could be something different than the | |
1025 | initial shift state and therefore the first character of the next line | |
1026 | is encoded using this state. But the state in question is never | |
7be8096f UD |
1027 | accessible to the user since the conversion stops after the NUL byte |
1028 | (which resets the state). Most stateful character sets in use today | |
1029 | require that the shift state after a newline is the initial state--but | |
1030 | this is not a strict guarantee. Therefore simply NUL terminating a | |
1031 | piece of a running text is not always an adequate solution and therefore | |
1032 | never should be used in generally used code. | |
390955cb | 1033 | |
61d97e92 | 1034 | The generic conversion interface (@pxref{Generic Charset Conversion}) |
d731df03 | 1035 | does not have this limitation (it simply works on buffers, not |
7be8096f | 1036 | strings), and the GNU C library contains a set of functions which take |
d731df03 UD |
1037 | additional parameters specifying the maximal number of bytes which are |
1038 | consumed from the input string. This way the problem of | |
1039 | @code{mbsrtowcs}'s example above could be solved by determining the line | |
1040 | length and passing this length to the function. | |
390955cb UD |
1041 | |
1042 | @comment wchar.h | |
1043 | @comment ISO | |
1044 | @deftypefun size_t wcsrtombs (char *restrict @var{dst}, const wchar_t **restrict @var{src}, size_t @var{len}, mbstate_t *restrict @var{ps}) | |
1045 | The @code{wcsrtombs} function (``wide character string restartable to | |
1046 | multibyte string'') converts the NUL terminated wide character string at | |
1047 | @code{*@var{src}} into an equivalent multibyte character string and | |
1048 | stores the result in the array pointed to by @var{dst}. The NUL wide | |
1049 | character is also converted. The conversion starts in the state | |
1050 | described in the object pointed to by @var{ps} or by a state object | |
1051 | locally to @code{wcsrtombs} in case @var{ps} is a null pointer. If | |
1052 | @var{dst} is a null pointer the conversion is performed as usual but the | |
1053 | result is not available. If all characters of the input string were | |
1054 | successfully converted and if @var{dst} is not a null pointer the | |
1055 | pointer pointed to by @var{src} gets assigned a null pointer. | |
1056 | ||
1057 | If one of the wide characters in the input string has no valid multibyte | |
1058 | character equivalent the conversion stops early, sets the global | |
1059 | variable @code{errno} to @code{EILSEQ}, and returns @code{(size_t) -1}. | |
1060 | ||
1061 | Another reason for a premature stop is if @var{dst} is not a null | |
1062 | pointer and the next converted character would require more than | |
1063 | @var{len} bytes in total to the array @var{dst}. In this case (and if | |
1064 | @var{dest} is not a null pointer) the pointer pointed to by @var{src} is | |
1065 | assigned a value pointing to the wide character right after the last one | |
1066 | successfully converted. | |
1067 | ||
1068 | Except in the case of an encoding error the return value of the function | |
1069 | is the number of bytes in all the multibyte character sequences stored | |
1070 | in @var{dst}. Before returning the state in the object pointed to by | |
1071 | @var{ps} (or the internal object in case @var{ps} is a null pointer) is | |
1072 | updated to reflect the state after the last conversion. The state is | |
1073 | the initial shift state in case the terminating NUL wide character was | |
1074 | converted. | |
1075 | ||
1076 | @pindex wchar.h | |
aaca11d8 | 1077 | This function was introduced in @w{Amendment 1} to @w{ISO C90} and is |
390955cb UD |
1078 | declared in @file{wchar.h}. |
1079 | @end deftypefun | |
1080 | ||
1081 | The restriction mentions above for the @code{mbsrtowcs} function applies | |
1082 | also here. There is no possibility to directly control the number of | |
1083 | input characters. One has to place the NUL wide character at the | |
1084 | correct place or control the consumed input indirectly via the available | |
1085 | output array size (the @var{len} parameter). | |
1086 | ||
1087 | @comment wchar.h | |
1088 | @comment GNU | |
1089 | @deftypefun size_t mbsnrtowcs (wchar_t *restrict @var{dst}, const char **restrict @var{src}, size_t @var{nmc}, size_t @var{len}, mbstate_t *restrict @var{ps}) | |
1090 | The @code{mbsnrtowcs} function is very similar to the @code{mbsrtowcs} | |
1091 | function. All the parameters are the same except for @var{nmc} which is | |
1092 | new. The return value is the same as for @code{mbsrtowcs}. | |
1093 | ||
1094 | This new parameter specifies how many bytes at most can be used from the | |
1095 | multibyte character string. I.e., the multibyte character string | |
1096 | @code{*@var{src}} need not be NUL terminated. But if a NUL byte is | |
1097 | found within the @var{nmc} first bytes of the string the conversion | |
1098 | stops here. | |
1099 | ||
1100 | This function is a GNU extensions. It is meant to work around the | |
1101 | problems mentioned above. Now it is possible to convert buffer with | |
1102 | multibyte character text piece for piece without having to care about | |
1103 | inserting NUL bytes and the effect of NUL bytes on the conversion state. | |
1104 | @end deftypefun | |
1105 | ||
1106 | A function to convert a multibyte string into a wide character string | |
d731df03 | 1107 | and display it could be written like this (this is not a really useful |
390955cb UD |
1108 | example): |
1109 | ||
1110 | @smallexample | |
1111 | void | |
1112 | showmbs (const char *src, FILE *fp) | |
1113 | @{ | |
1114 | mbstate_t state; | |
1115 | int cnt = 0; | |
1116 | memset (&state, '\0', sizeof (state)); | |
1117 | while (1) | |
1118 | @{ | |
1119 | wchar_t linebuf[100]; | |
1120 | const char *endp = strchr (src, '\n'); | |
1121 | size_t n; | |
1122 | ||
1123 | /* @r{Exit if there is no more line.} */ | |
1124 | if (endp == NULL) | |
1125 | break; | |
1126 | ||
1127 | n = mbsnrtowcs (linebuf, &src, endp - src, 99, &state); | |
1128 | linebuf[n] = L'\0'; | |
1129 | fprintf (fp, "line %d: \"%S\"\n", linebuf); | |
1130 | @} | |
1131 | @} | |
1132 | @end smallexample | |
1133 | ||
d731df03 UD |
1134 | There is no problem with the state after a call to @code{mbsnrtowcs}. |
1135 | Since we don't insert characters in the strings which were not in there | |
1136 | right from the beginning and we use @var{state} only for the conversion | |
1137 | of the given buffer there is no problem with altering the state. | |
390955cb UD |
1138 | |
1139 | @comment wchar.h | |
1140 | @comment GNU | |
1141 | @deftypefun size_t wcsnrtombs (char *restrict @var{dst}, const wchar_t **restrict @var{src}, size_t @var{nwc}, size_t @var{len}, mbstate_t *restrict @var{ps}) | |
1142 | The @code{wcsnrtombs} function implements the conversion from wide | |
1143 | character strings to multibyte character strings. It is similar to | |
1144 | @code{wcsrtombs} but it takes, just like @code{mbsnrtowcs}, an extra | |
1145 | parameter which specifies the length of the input string. | |
1146 | ||
1147 | No more than @var{nwc} wide characters from the input string | |
1148 | @code{*@var{src}} are converted. If the input string contains a NUL | |
1149 | wide character in the first @var{nwc} character to conversion stops at | |
1150 | this place. | |
1151 | ||
1152 | This function is a GNU extension and just like @code{mbsnrtowcs} is | |
1153 | helps in situations where no NUL terminated input strings are available. | |
1154 | @end deftypefun | |
1155 | ||
1156 | ||
1157 | @node Multibyte Conversion Example | |
1158 | @subsection A Complete Multibyte Conversion Example | |
1159 | ||
1160 | The example programs given in the last sections are only brief and do | |
d731df03 | 1161 | not contain all the error checking etc. Presented here is a complete |
390955cb UD |
1162 | and documented example. It features the @code{mbrtowc} function but it |
1163 | should be easy to derive versions using the other functions. | |
1164 | ||
1165 | @smallexample | |
1166 | int | |
1167 | file_mbsrtowcs (int input, int output) | |
1168 | @{ | |
1169 | /* @r{Note the use of @code{MB_LEN_MAX}.} | |
1170 | @r{@code{MB_CUR_MAX} cannot portably be used here.} */ | |
1171 | char buffer[BUFSIZ + MB_LEN_MAX]; | |
1172 | mbstate_t state; | |
1173 | int filled = 0; | |
1174 | int eof = 0; | |
1175 | ||
1176 | /* @r{Initialize the state.} */ | |
1177 | memset (&state, '\0', sizeof (state)); | |
1178 | ||
1179 | while (!eof) | |
1180 | @{ | |
1181 | ssize_t nread; | |
1182 | ssize_t nwrite; | |
1183 | char *inp = buffer; | |
1184 | wchar_t outbuf[BUFSIZ]; | |
1185 | wchar_t *outp = outbuf; | |
1186 | ||
1187 | /* @r{Fill up the buffer from the input file.} */ | |
1188 | nread = read (input, buffer + filled, BUFSIZ); | |
1189 | if (nread < 0) | |
1190 | @{ | |
1191 | perror ("read"); | |
1192 | return 0; | |
1193 | @} | |
1194 | /* @r{If we reach end of file, make a note to read no more.} */ | |
1195 | if (nread == 0) | |
1196 | eof = 1; | |
1197 | ||
1198 | /* @r{@code{filled} is now the number of bytes in @code{buffer}.} */ | |
1199 | filled += nread; | |
1200 | ||
1201 | /* @r{Convert those bytes to wide characters--as many as we can.} */ | |
1202 | while (1) | |
1203 | @{ | |
1204 | size_t thislen = mbrtowc (outp, inp, filled, &state); | |
1205 | /* @r{Stop converting at invalid character;} | |
1206 | @r{this can mean we have read just the first part} | |
1207 | @r{of a valid character.} */ | |
1208 | if (thislen == (size_t) -1) | |
1209 | break; | |
1210 | /* @r{We want to handle embedded NUL bytes} | |
1211 | @r{but the return value is 0. Correct this.} */ | |
1212 | if (thislen == 0) | |
1213 | thislen = 1; | |
1214 | /* @r{Advance past this character.} */ | |
1215 | inp += thislen; | |
1216 | filled -= thislen; | |
1217 | ++outp; | |
1218 | @} | |
1219 | ||
1220 | /* @r{Write the wide characters we just made.} */ | |
1221 | nwrite = write (output, outbuf, | |
1222 | (outp - outbuf) * sizeof (wchar_t)); | |
1223 | if (nwrite < 0) | |
1224 | @{ | |
1225 | perror ("write"); | |
1226 | return 0; | |
1227 | @} | |
1228 | ||
1229 | /* @r{See if we have a @emph{real} invalid character.} */ | |
1230 | if ((eof && filled > 0) || filled >= MB_CUR_MAX) | |
1231 | @{ | |
1232 | error (0, 0, "invalid multibyte character"); | |
1233 | return 0; | |
1234 | @} | |
1235 | ||
1236 | /* @r{If any characters must be carried forward,} | |
1237 | @r{put them at the beginning of @code{buffer}.} */ | |
1238 | if (filled > 0) | |
1239 | memmove (inp, buffer, filled); | |
1240 | @} | |
1241 | ||
1242 | return 1; | |
1243 | @} | |
1244 | @end smallexample | |
1245 | ||
1246 | ||
1247 | @node Non-reentrant Conversion | |
1248 | @section Non-reentrant Conversion Function | |
1249 | ||
aaca11d8 UD |
1250 | The functions described in the last chapter are defined in |
1251 | @w{Amendment 1} to @w{ISO C90}. But the original @w{ISO C90} standard also | |
390955cb UD |
1252 | contained functions for character set conversion. The reason that they |
1253 | are not described in the first place is that they are almost entirely | |
1254 | useless. | |
1255 | ||
1256 | The problem is that all the functions for conversion defined in @w{ISO | |
976780fd | 1257 | C90} use a local state. This implies that multiple conversions at the |
d731df03 UD |
1258 | same time (not only when using threads) cannot be done, and that you |
1259 | cannot first convert single characters and then strings since you cannot | |
1260 | tell the conversion functions which state to use. | |
390955cb UD |
1261 | |
1262 | These functions are therefore usable only in a very limited set of | |
7be8096f | 1263 | situations. One must complete converting the entire string before |
390955cb UD |
1264 | starting a new one and each string/text must be converted with the same |
1265 | function (there is no problem with the library itself; it is guaranteed | |
1266 | that no library function changes the state of any of these functions). | |
d731df03 UD |
1267 | @strong{For the above reasons it is highly requested that the functions |
1268 | from the last section are used in place of non-reentrant conversion | |
1269 | functions.} | |
390955cb UD |
1270 | |
1271 | @menu | |
1272 | * Non-reentrant Character Conversion:: Non-reentrant Conversion of Single | |
1273 | Characters. | |
1274 | * Non-reentrant String Conversion:: Non-reentrant Conversion of Strings. | |
1275 | * Shift State:: States in Non-reentrant Functions. | |
1276 | @end menu | |
1277 | ||
1278 | @node Non-reentrant Character Conversion | |
1279 | @subsection Non-reentrant Conversion of Single Characters | |
1280 | ||
1281 | @comment stdlib.h | |
1282 | @comment ISO | |
7be8096f | 1283 | @deftypefun int mbtowc (wchar_t *restrict @var{result}, const char *restrict @var{string}, size_t @var{size}) |
390955cb UD |
1284 | The @code{mbtowc} (``multibyte to wide character'') function when called |
1285 | with non-null @var{string} converts the first multibyte character | |
1286 | beginning at @var{string} to its corresponding wide character code. It | |
1287 | stores the result in @code{*@var{result}}. | |
1288 | ||
1289 | @code{mbtowc} never examines more than @var{size} bytes. (The idea is | |
1290 | to supply for @var{size} the number of bytes of data you have in hand.) | |
1291 | ||
1292 | @code{mbtowc} with non-null @var{string} distinguishes three | |
1293 | possibilities: the first @var{size} bytes at @var{string} start with | |
1294 | valid multibyte character, they start with an invalid byte sequence or | |
1295 | just part of a character, or @var{string} points to an empty string (a | |
1296 | null character). | |
1297 | ||
1298 | For a valid multibyte character, @code{mbtowc} converts it to a wide | |
1299 | character and stores that in @code{*@var{result}}, and returns the | |
7be8096f | 1300 | number of bytes in that character (always at least @math{1}, and never |
390955cb UD |
1301 | more than @var{size}). |
1302 | ||
7be8096f UD |
1303 | For an invalid byte sequence, @code{mbtowc} returns @math{-1}. For an |
1304 | empty string, it returns @math{0}, also storing @code{'\0'} in | |
390955cb UD |
1305 | @code{*@var{result}}. |
1306 | ||
1307 | If the multibyte character code uses shift characters, then | |
1308 | @code{mbtowc} maintains and updates a shift state as it scans. If you | |
1309 | call @code{mbtowc} with a null pointer for @var{string}, that | |
1310 | initializes the shift state to its standard initial value. It also | |
1311 | returns nonzero if the multibyte character code in use actually has a | |
1312 | shift state. @xref{Shift State}. | |
1313 | @end deftypefun | |
1314 | ||
1315 | @comment stdlib.h | |
1316 | @comment ISO | |
1317 | @deftypefun int wctomb (char *@var{string}, wchar_t @var{wchar}) | |
1318 | The @code{wctomb} (``wide character to multibyte'') function converts | |
1319 | the wide character code @var{wchar} to its corresponding multibyte | |
1320 | character sequence, and stores the result in bytes starting at | |
1321 | @var{string}. At most @code{MB_CUR_MAX} characters are stored. | |
1322 | ||
1323 | @code{wctomb} with non-null @var{string} distinguishes three | |
1324 | possibilities for @var{wchar}: a valid wide character code (one that can | |
7be8096f | 1325 | be translated to a multibyte character), an invalid code, and @code{L'\0'}. |
390955cb UD |
1326 | |
1327 | Given a valid code, @code{wctomb} converts it to a multibyte character, | |
1328 | storing the bytes starting at @var{string}. Then it returns the number | |
7be8096f | 1329 | of bytes in that character (always at least @math{1}, and never more |
390955cb UD |
1330 | than @code{MB_CUR_MAX}). |
1331 | ||
1332 | If @var{wchar} is an invalid wide character code, @code{wctomb} returns | |
7be8096f UD |
1333 | @math{-1}. If @var{wchar} is @code{L'\0'}, it returns @code{0}, also |
1334 | storing @code{'\0'} in @code{*@var{string}}. | |
390955cb UD |
1335 | |
1336 | If the multibyte character code uses shift characters, then | |
1337 | @code{wctomb} maintains and updates a shift state as it scans. If you | |
1338 | call @code{wctomb} with a null pointer for @var{string}, that | |
1339 | initializes the shift state to its standard initial value. It also | |
1340 | returns nonzero if the multibyte character code in use actually has a | |
1341 | shift state. @xref{Shift State}. | |
1342 | ||
1343 | Calling this function with a @var{wchar} argument of zero when | |
1344 | @var{string} is not null has the side-effect of reinitializing the | |
1345 | stored shift state @emph{as well as} storing the multibyte character | |
7be8096f | 1346 | @code{'\0'} and returning @math{0}. |
390955cb UD |
1347 | @end deftypefun |
1348 | ||
1349 | Similar to @code{mbrlen} there is also a non-reentrant function which | |
1350 | computes the length of a multibyte character. It can be defined in | |
1351 | terms of @code{mbtowc}. | |
1352 | ||
1353 | @comment stdlib.h | |
1354 | @comment ISO | |
1355 | @deftypefun int mblen (const char *@var{string}, size_t @var{size}) | |
1356 | The @code{mblen} function with a non-null @var{string} argument returns | |
1357 | the number of bytes that make up the multibyte character beginning at | |
1358 | @var{string}, never examining more than @var{size} bytes. (The idea is | |
1359 | to supply for @var{size} the number of bytes of data you have in hand.) | |
1360 | ||
1361 | The return value of @code{mblen} distinguishes three possibilities: the | |
1362 | first @var{size} bytes at @var{string} start with valid multibyte | |
1363 | character, they start with an invalid byte sequence or just part of a | |
1364 | character, or @var{string} points to an empty string (a null character). | |
1365 | ||
1366 | For a valid multibyte character, @code{mblen} returns the number of | |
1367 | bytes in that character (always at least @code{1}, and never more than | |
1368 | @var{size}). For an invalid byte sequence, @code{mblen} returns | |
7be8096f | 1369 | @math{-1}. For an empty string, it returns @math{0}. |
390955cb UD |
1370 | |
1371 | If the multibyte character code uses shift characters, then @code{mblen} | |
1372 | maintains and updates a shift state as it scans. If you call | |
1373 | @code{mblen} with a null pointer for @var{string}, that initializes the | |
7be8096f UD |
1374 | shift state to its standard initial value. It also returns a nonzero |
1375 | value if the multibyte character code in use actually has a shift state. | |
390955cb UD |
1376 | @xref{Shift State}. |
1377 | ||
1378 | @pindex stdlib.h | |
1379 | The function @code{mblen} is declared in @file{stdlib.h}. | |
1380 | @end deftypefun | |
1381 | ||
1382 | ||
1383 | @node Non-reentrant String Conversion | |
1384 | @subsection Non-reentrant Conversion of Strings | |
1385 | ||
976780fd | 1386 | For convenience reasons the @w{ISO C90} standard defines also functions |
390955cb | 1387 | to convert entire strings instead of single characters. These functions |
aaca11d8 UD |
1388 | suffer from the same problems as their reentrant counterparts from |
1389 | @w{Amendment 1} to @w{ISO C90}; see @ref{Converting Strings}. | |
390955cb UD |
1390 | |
1391 | @comment stdlib.h | |
1392 | @comment ISO | |
1393 | @deftypefun size_t mbstowcs (wchar_t *@var{wstring}, const char *@var{string}, size_t @var{size}) | |
1394 | The @code{mbstowcs} (``multibyte string to wide character string'') | |
1395 | function converts the null-terminated string of multibyte characters | |
1396 | @var{string} to an array of wide character codes, storing not more than | |
1397 | @var{size} wide characters into the array beginning at @var{wstring}. | |
1398 | The terminating null character counts towards the size, so if @var{size} | |
1399 | is less than the actual number of wide characters resulting from | |
1400 | @var{string}, no terminating null character is stored. | |
1401 | ||
1402 | The conversion of characters from @var{string} begins in the initial | |
1403 | shift state. | |
1404 | ||
1405 | If an invalid multibyte character sequence is found, this function | |
7be8096f | 1406 | returns a value of @math{-1}. Otherwise, it returns the number of wide |
390955cb UD |
1407 | characters stored in the array @var{wstring}. This number does not |
1408 | include the terminating null character, which is present if the number | |
1409 | is less than @var{size}. | |
1410 | ||
1411 | Here is an example showing how to convert a string of multibyte | |
1412 | characters, allocating enough space for the result. | |
1413 | ||
1414 | @smallexample | |
1415 | wchar_t * | |
1416 | mbstowcs_alloc (const char *string) | |
1417 | @{ | |
1418 | size_t size = strlen (string) + 1; | |
1419 | wchar_t *buf = xmalloc (size * sizeof (wchar_t)); | |
1420 | ||
1421 | size = mbstowcs (buf, string, size); | |
1422 | if (size == (size_t) -1) | |
1423 | return NULL; | |
1424 | buf = xrealloc (buf, (size + 1) * sizeof (wchar_t)); | |
1425 | return buf; | |
1426 | @} | |
1427 | @end smallexample | |
1428 | ||
1429 | @end deftypefun | |
1430 | ||
1431 | @comment stdlib.h | |
1432 | @comment ISO | |
1433 | @deftypefun size_t wcstombs (char *@var{string}, const wchar_t *@var{wstring}, size_t @var{size}) | |
1434 | The @code{wcstombs} (``wide character string to multibyte string'') | |
1435 | function converts the null-terminated wide character array @var{wstring} | |
1436 | into a string containing multibyte characters, storing not more than | |
1437 | @var{size} bytes starting at @var{string}, followed by a terminating | |
1438 | null character if there is room. The conversion of characters begins in | |
1439 | the initial shift state. | |
1440 | ||
1441 | The terminating null character counts towards the size, so if @var{size} | |
1442 | is less than or equal to the number of bytes needed in @var{wstring}, no | |
1443 | terminating null character is stored. | |
1444 | ||
1445 | If a code that does not correspond to a valid multibyte character is | |
7be8096f | 1446 | found, this function returns a value of @math{-1}. Otherwise, the |
390955cb UD |
1447 | return value is the number of bytes stored in the array @var{string}. |
1448 | This number does not include the terminating null character, which is | |
1449 | present if the number is less than @var{size}. | |
1450 | @end deftypefun | |
1451 | ||
1452 | @node Shift State | |
1453 | @subsection States in Non-reentrant Functions | |
1454 | ||
1455 | In some multibyte character codes, the @emph{meaning} of any particular | |
1456 | byte sequence is not fixed; it depends on what other sequences have come | |
1457 | earlier in the same string. Typically there are just a few sequences | |
1458 | that can change the meaning of other sequences; these few are called | |
1459 | @dfn{shift sequences} and we say that they set the @dfn{shift state} for | |
1460 | other sequences that follow. | |
1461 | ||
1462 | To illustrate shift state and shift sequences, suppose we decide that | |
1463 | the sequence @code{0200} (just one byte) enters Japanese mode, in which | |
1464 | pairs of bytes in the range from @code{0240} to @code{0377} are single | |
1465 | characters, while @code{0201} enters Latin-1 mode, in which single bytes | |
1466 | in the range from @code{0240} to @code{0377} are characters, and | |
1467 | interpreted according to the ISO Latin-1 character set. This is a | |
1468 | multibyte code which has two alternative shift states (``Japanese mode'' | |
1469 | and ``Latin-1 mode''), and two shift sequences that specify particular | |
1470 | shift states. | |
1471 | ||
1472 | When the multibyte character code in use has shift states, then | |
1473 | @code{mblen}, @code{mbtowc} and @code{wctomb} must maintain and update | |
1474 | the current shift state as they scan the string. To make this work | |
1475 | properly, you must follow these rules: | |
1476 | ||
1477 | @itemize @bullet | |
1478 | @item | |
1479 | Before starting to scan a string, call the function with a null pointer | |
1480 | for the multibyte character address---for example, @code{mblen (NULL, | |
1481 | 0)}. This initializes the shift state to its standard initial value. | |
1482 | ||
1483 | @item | |
1484 | Scan the string one character at a time, in order. Do not ``back up'' | |
1485 | and rescan characters already scanned, and do not intersperse the | |
1486 | processing of different strings. | |
1487 | @end itemize | |
1488 | ||
1489 | Here is an example of using @code{mblen} following these rules: | |
1490 | ||
1491 | @smallexample | |
1492 | void | |
1493 | scan_string (char *s) | |
1494 | @{ | |
1495 | int length = strlen (s); | |
1496 | ||
d731df03 | 1497 | /* @r{Initialize shift state.} */ |
390955cb UD |
1498 | mblen (NULL, 0); |
1499 | ||
1500 | while (1) | |
1501 | @{ | |
1502 | int thischar = mblen (s, length); | |
d731df03 | 1503 | /* @r{Deal with end of string and invalid characters.} */ |
390955cb UD |
1504 | if (thischar == 0) |
1505 | break; | |
1506 | if (thischar == -1) | |
1507 | @{ | |
1508 | error ("invalid multibyte character"); | |
1509 | break; | |
1510 | @} | |
d731df03 | 1511 | /* @r{Advance past this character.} */ |
390955cb UD |
1512 | s += thischar; |
1513 | length -= thischar; | |
1514 | @} | |
1515 | @} | |
1516 | @end smallexample | |
1517 | ||
1518 | The functions @code{mblen}, @code{mbtowc} and @code{wctomb} are not | |
1519 | reentrant when using a multibyte code that uses a shift state. However, | |
1520 | no other library functions call these functions, so you don't have to | |
1521 | worry that the shift state will be changed mysteriously. | |
1522 | ||
1523 | ||
1524 | @node Generic Charset Conversion | |
1525 | @section Generic Charset Conversion | |
1526 | ||
1527 | The conversion functions mentioned so far in this chapter all had in | |
1528 | common that they operate on character sets which are not directly | |
1529 | specified by the functions. The multibyte encoding used is specified by | |
1530 | the currently selected locale for the @code{LC_CTYPE} category. The | |
1531 | wide character set is fixed by the implementation (in the case of GNU C | |
aaca11d8 | 1532 | library it always is UCS-4 encoded @w{ISO 10646}. |
390955cb UD |
1533 | |
1534 | This has of course several problems when it comes to general character | |
1535 | conversion: | |
1536 | ||
1537 | @itemize @bullet | |
1538 | @item | |
1539 | For every conversion where neither the source or destination character | |
1540 | set is the character set of the locale for the @code{LC_CTYPE} category, | |
1541 | one has to change the @code{LC_CTYPE} locale using @code{setlocale}. | |
1542 | ||
1543 | This introduces major problems for the rest of the programs since | |
1544 | several more functions (e.g., the character classification functions, | |
464d646f | 1545 | @pxref{Classification of Characters}) use the @code{LC_CTYPE} category. |
390955cb UD |
1546 | |
1547 | @item | |
1548 | Parallel conversions to and from different character sets are not | |
1549 | possible since the @code{LC_CTYPE} selection is global and shared by all | |
1550 | threads. | |
1551 | ||
1552 | @item | |
1553 | If neither the source nor the destination character set is the character | |
1554 | set used for @code{wchar_t} representation there is at least a two-step | |
1555 | process necessary to convert a text using the functions above. One | |
1556 | would have to select the source character set as the multibyte encoding, | |
1557 | convert the text into a @code{wchar_t} text, select the destination | |
1558 | character set as the multibyte encoding and convert the wide character | |
7be8096f | 1559 | text to the multibyte (@math{=} destination) character set. |
390955cb UD |
1560 | |
1561 | Even if this is possible (which is not guaranteed) it is a very tiring | |
1562 | work. Plus it suffers from the other two raised points even more due to | |
1563 | the steady changing of the locale. | |
1564 | @end itemize | |
1565 | ||
1566 | ||
1567 | The XPG2 standard defines a completely new set of functions which has | |
1568 | none of these limitations. They are not at all coupled to the selected | |
1569 | locales and they but no constraints on the character sets selected for | |
1570 | source and destination. Only the set of available conversions is | |
1571 | limiting them. The standard does not specify that any conversion at all | |
1572 | must be available. It is a measure of the quality of the implementation. | |
1573 | ||
d731df03 UD |
1574 | In the following text first the interface to @code{iconv}, the |
1575 | conversion function, will be described. Comparisons with other | |
1576 | implementations will show what pitfalls lie on the way of portable | |
1577 | applications. At last, the implementation is described as far as | |
1578 | interesting to the advanced user who wants to extend the conversion | |
1579 | capabilities. | |
390955cb UD |
1580 | |
1581 | @menu | |
1582 | * Generic Conversion Interface:: Generic Character Set Conversion Interface. | |
1583 | * iconv Examples:: A complete @code{iconv} example. | |
1584 | * Other iconv Implementations:: Some Details about other @code{iconv} | |
1585 | Implementations. | |
1586 | * glibc iconv Implementation:: The @code{iconv} Implementation in the GNU C | |
1587 | library. | |
1588 | @end menu | |
1589 | ||
1590 | @node Generic Conversion Interface | |
1591 | @subsection Generic Character Set Conversion Interface | |
1592 | ||
1593 | This set of functions follows the traditional cycle of using a resource: | |
1594 | open--use--close. The interface consists of three functions, each of | |
1595 | which implement one step. | |
1596 | ||
1597 | Before the interfaces are described it is necessary to introduce a | |
1598 | datatype. Just like other open--use--close interface the functions | |
1599 | introduced here work using a handles and the @file{iconv.h} header | |
1600 | defines a special type for the handles used. | |
1601 | ||
1602 | @comment iconv.h | |
1603 | @comment XPG2 | |
1604 | @deftp {Data Type} iconv_t | |
1605 | This data type is an abstract type defined in @file{iconv.h}. The user | |
1606 | must not assume anything about the definition of this type, it must be | |
1607 | completely opaque. | |
1608 | ||
1609 | Objects of this type can get assigned handles for the conversions using | |
1610 | the @code{iconv} functions. The objects themselves need not be freed but | |
1611 | the conversions for which the handles stand for have to. | |
1612 | @end deftp | |
1613 | ||
1614 | @noindent | |
1615 | The first step is the function to create a handle. | |
1616 | ||
1617 | @comment iconv.h | |
1618 | @comment XPG2 | |
1619 | @deftypefun iconv_t iconv_open (const char *@var{tocode}, const char *@var{fromcode}) | |
1620 | The @code{iconv_open} function has to be used before starting a | |
1621 | conversion. The two parameters this function takes determine the | |
c1b2d472 | 1622 | source and destination character set for the conversion and if the |
390955cb UD |
1623 | implementation has the possibility to perform such a conversion the |
1624 | function returns a handle. | |
1625 | ||
1626 | If the wanted conversion is not available the function returns | |
1627 | @code{(iconv_t) -1}. In this case the global variable @code{errno} can | |
1628 | have the following values: | |
1629 | ||
1630 | @table @code | |
1631 | @item EMFILE | |
1632 | The process already has @code{OPEN_MAX} file descriptors open. | |
1633 | @item ENFILE | |
1634 | The system limit of open file is reached. | |
1635 | @item ENOMEM | |
1636 | Not enough memory to carry out the operation. | |
1637 | @item EINVAL | |
1638 | The conversion from @var{fromcode} to @var{tocode} is not supported. | |
1639 | @end table | |
1640 | ||
1641 | It is not possible to use the same descriptor in different threads to | |
1642 | perform independent conversions. Within the data structures associated | |
1643 | with the descriptor there is information about the conversion state. | |
d731df03 | 1644 | This must not be messed up by using it in different conversions. |
390955cb | 1645 | |
c1b2d472 | 1646 | An @code{iconv} descriptor is like a file descriptor as for every use a |
390955cb UD |
1647 | new descriptor must be created. The descriptor does not stand for all |
1648 | of the conversions from @var{fromset} to @var{toset}. | |
1649 | ||
1650 | The GNU C library implementation of @code{iconv_open} has one | |
1651 | significant extension to other implementations. To ease the extension | |
49c091e5 UD |
1652 | of the set of available conversions the implementation allows storing |
1653 | the necessary files with data and code in arbitrarily many directories. | |
1654 | How this extension has to be written will be explained below | |
390955cb UD |
1655 | (@pxref{glibc iconv Implementation}). Here it is only important to say |
1656 | that all directories mentioned in the @code{GCONV_PATH} environment | |
1657 | variable are considered if they contain a file @file{gconv-modules}. | |
1658 | These directories need not necessarily be created by the system | |
1659 | administrator. In fact, this extension is introduced to help users | |
49c091e5 | 1660 | writing and using their own, new conversions. Of course this does not work |
390955cb UD |
1661 | for security reasons in SUID binaries; in this case only the system |
1662 | directory is considered and this normally is | |
1663 | @file{@var{prefix}/lib/gconv}. The @code{GCONV_PATH} environment | |
1664 | variable is examined exactly once at the first call of the | |
1665 | @code{iconv_open} function. Later modifications of the variable have no | |
1666 | effect. | |
1667 | ||
1668 | @pindex iconv.h | |
1669 | This function got introduced early in the X/Open Portability Guide, | |
1670 | @w{version 2}. It is supported by all commercial Unices as it is | |
d731df03 UD |
1671 | required for the Unix branding. However, the quality and completeness |
1672 | of the implementation varies widely. The function is declared in | |
390955cb UD |
1673 | @file{iconv.h}. |
1674 | @end deftypefun | |
1675 | ||
1676 | The @code{iconv} implementation can associate large data structure with | |
1677 | the handle returned by @code{iconv_open}. Therefore it is crucial to | |
1678 | free all the resources once all conversions are carried out and the | |
1679 | conversion is not needed anymore. | |
1680 | ||
1681 | @comment iconv.h | |
1682 | @comment XPG2 | |
1683 | @deftypefun int iconv_close (iconv_t @var{cd}) | |
1684 | The @code{iconv_close} function frees all resources associated with the | |
1685 | handle @var{cd} which must have been returned by a successful call to | |
1686 | the @code{iconv_open} function. | |
1687 | ||
1688 | If the function call was successful the return value is @math{0}. | |
1689 | Otherwise it is @math{-1} and @code{errno} is set appropriately. | |
1690 | Defined error are: | |
1691 | ||
1692 | @table @code | |
1693 | @item EBADF | |
1694 | The conversion descriptor is invalid. | |
1695 | @end table | |
1696 | ||
1697 | @pindex iconv.h | |
1698 | This function was introduced together with the rest of the @code{iconv} | |
1699 | functions in XPG2 and it is declared in @file{iconv.h}. | |
1700 | @end deftypefun | |
1701 | ||
1702 | The standard defines only one actual conversion function. This has | |
1703 | therefore the most general interface: it allows conversion from one | |
1704 | buffer to another. Conversion from a file to a buffer, vice versa, or | |
1705 | even file to file can be implemented on top of it. | |
1706 | ||
1707 | @comment iconv.h | |
1708 | @comment XPG2 | |
70a1a465 | 1709 | @deftypefun size_t iconv (iconv_t @var{cd}, char **@var{inbuf}, size_t *@var{inbytesleft}, char **@var{outbuf}, size_t *@var{outbytesleft}) |
390955cb UD |
1710 | @cindex stateful |
1711 | The @code{iconv} function converts the text in the input buffer | |
1712 | according to the rules associated with the descriptor @var{cd} and | |
1713 | stores the result in the output buffer. It is possible to call the | |
1714 | function for the same text several times in a row since for stateful | |
1715 | character sets the necessary state information is kept in the data | |
1716 | structures associated with the descriptor. | |
1717 | ||
1718 | The input buffer is specified by @code{*@var{inbuf}} and it contains | |
1719 | @code{*@var{inbytesleft}} bytes. The extra indirection is necessary for | |
1720 | communicating the used input back to the caller (see below). It is | |
1721 | important to note that the buffer pointer is of type @code{char} and the | |
1722 | length is measured in bytes even if the input text is encoded in wide | |
1723 | characters. | |
1724 | ||
1725 | The output buffer is specified in a similar way. @code{*@var{outbuf}} | |
1726 | points to the beginning of the buffer with at least | |
1727 | @code{*@var{outbytesleft}} bytes room for the result. The buffer | |
1728 | pointer again is of type @code{char} and the length is measured in | |
1729 | bytes. If @var{outbuf} or @code{*@var{outbuf}} is a null pointer the | |
1730 | conversion is performed but no output is available. | |
1731 | ||
1732 | If @var{inbuf} is a null pointer the @code{iconv} function performs the | |
1733 | necessary action to put the state of the conversion into the initial | |
1734 | state. This is obviously a no-op for non-stateful encodings, but if the | |
1735 | encoding has a state such a function call might put some byte sequences | |
1736 | in the output buffer which perform the necessary state changes. The | |
1737 | next call with @var{inbuf} not being a null pointer then simply goes on | |
1738 | from the initial state. It is important that the programmer never makes | |
1739 | any assumption on whether the conversion has to deal with states or not. | |
1740 | Even if the input and output character sets are not stateful the | |
1741 | implementation might still have to keep states. This is due to the | |
1742 | implementation chosen for the GNU C library as it is described below. | |
1743 | Therefore an @code{iconv} call to reset the state should always be | |
1744 | performed if some protocol requires this for the output text. | |
1745 | ||
1746 | The conversion stops for three reasons. The first is that all | |
1747 | characters from the input buffer are converted. This actually can mean | |
c1b2d472 | 1748 | two things: really all bytes from the input buffer are consumed or |
390955cb UD |
1749 | there are some bytes at the end of the buffer which possibly can form a |
1750 | complete character but the input is incomplete. The second reason for a | |
1751 | stop is when the output buffer is full. And the third reason is that | |
1752 | the input contains invalid characters. | |
1753 | ||
1754 | In all these cases the buffer pointers after the last successful | |
1755 | conversion, for input and output buffer, are stored in @var{inbuf} and | |
1756 | @var{outbuf} and the available room in each buffer is stored in | |
1757 | @var{inbytesleft} and @var{outbytesleft}. | |
1758 | ||
1759 | Since the character sets selected in the @code{iconv_open} call can be | |
1760 | almost arbitrary there can be situations where the input buffer contains | |
1761 | valid characters which have no identical representation in the output | |
1762 | character set. The behavior in this situation is undefined. The | |
1763 | @emph{current} behavior of the GNU C library in this situation is to | |
1764 | return with an error immediately. This certainly is not the most | |
1765 | desirable solution. Therefore future versions will provide better ones | |
1766 | but they are not yet finished. | |
1767 | ||
1768 | If all input from the input buffer is successfully converted and stored | |
e3a7ff87 UD |
1769 | in the output buffer the function returns the number of non-reversible |
1770 | conversions performed. In all other cases the return value is | |
1771 | @code{(size_t) -1} and @code{errno} is set appropriately. In this case | |
1772 | the value pointed to by @var{inbytesleft} is nonzero. | |
390955cb UD |
1773 | |
1774 | @table @code | |
1775 | @item EILSEQ | |
1776 | The conversion stopped because of an invalid byte sequence in the input. | |
1777 | After the call @code{*@var{inbuf}} points at the first byte of the | |
1778 | invalid byte sequence. | |
1779 | ||
1780 | @item E2BIG | |
1781 | The conversion stopped because it ran out of space in the output buffer. | |
1782 | ||
1783 | @item EINVAL | |
1784 | The conversion stopped because of an incomplete byte sequence at the end | |
1785 | of the input buffer. | |
1786 | ||
1787 | @item EBADF | |
1788 | The @var{cd} argument is invalid. | |
1789 | @end table | |
1790 | ||
1791 | @pindex iconv.h | |
1792 | This function was introduced in the XPG2 standard and is declared in the | |
1793 | @file{iconv.h} header. | |
1794 | @end deftypefun | |
1795 | ||
1796 | The definition of the @code{iconv} function is quite good overall. It | |
1797 | provides quite flexible functionality. The only problems lie in the | |
1798 | boundary cases which are incomplete byte sequences at the end of the | |
d731df03 UD |
1799 | input buffer and invalid input. A third problem, which is not really |
1800 | a design problem, is the way conversions are selected. The standard | |
1801 | does not say anything about the legitimate names, a minimal set of | |
1802 | available conversions. We will see how this negatively impacts other | |
1803 | implementations, as is demonstrated below. | |
390955cb UD |
1804 | |
1805 | ||
1806 | @node iconv Examples | |
1807 | @subsection A complete @code{iconv} example | |
1808 | ||
1809 | The example below features a solution for a common problem. Given that | |
1810 | one knows the internal encoding used by the system for @code{wchar_t} | |
1811 | strings one often is in the position to read text from a file and store | |
1812 | it in wide character buffers. One can do this using @code{mbsrtowcs} | |
1813 | but then we run into the problems discussed above. | |
1814 | ||
1815 | @smallexample | |
1816 | int | |
1817 | file2wcs (int fd, const char *charset, wchar_t *outbuf, size_t avail) | |
1818 | @{ | |
1819 | char inbuf[BUFSIZ]; | |
1820 | size_t insize = 0; | |
1821 | char *wrptr = (char *) outbuf; | |
1822 | int result = 0; | |
1823 | iconv_t cd; | |
1824 | ||
e0ea3583 | 1825 | cd = iconv_open ("WCHAR_T", charset); |
390955cb UD |
1826 | if (cd == (iconv_t) -1) |
1827 | @{ | |
1828 | /* @r{Something went wrong.} */ | |
1829 | if (errno == EINVAL) | |
e0ea3583 | 1830 | error (0, 0, "conversion from '%s' to wchar_t not available", |
390955cb UD |
1831 | charset); |
1832 | else | |
1833 | perror ("iconv_open"); | |
1834 | ||
1835 | /* @r{Terminate the output string.} */ | |
1836 | *outbuf = L'\0'; | |
1837 | ||
1838 | return -1; | |
1839 | @} | |
1840 | ||
1841 | while (avail > 0) | |
1842 | @{ | |
1843 | size_t nread; | |
1844 | size_t nconv; | |
1845 | char *inptr = inbuf; | |
1846 | ||
1847 | /* @r{Read more input.} */ | |
1848 | nread = read (fd, inbuf + insize, sizeof (inbuf) - insize); | |
1849 | if (nread == 0) | |
1850 | @{ | |
1851 | /* @r{When we come here the file is completely read.} | |
1852 | @r{This still could mean there are some unused} | |
1853 | @r{characters in the @code{inbuf}. Put them back.} */ | |
1854 | if (lseek (fd, -insize, SEEK_CUR) == -1) | |
1855 | result = -1; | |
c559a3ca UD |
1856 | |
1857 | /* @r{Now write out the byte sequence to get into the} | |
1858 | @r{initial state if this is necessary.} */ | |
1859 | iconv (cd, NULL, NULL, &wrptr, &avail); | |
1860 | ||
390955cb UD |
1861 | break; |
1862 | @} | |
1863 | insize += nread; | |
1864 | ||
1865 | /* @r{Do the conversion.} */ | |
1866 | nconv = iconv (cd, &inptr, &insize, &wrptr, &avail); | |
1867 | if (nconv == (size_t) -1) | |
1868 | @{ | |
1869 | /* @r{Not everything went right. It might only be} | |
1870 | @r{an unfinished byte sequence at the end of the} | |
1871 | @r{buffer. Or it is a real problem.} */ | |
1872 | if (errno == EINVAL) | |
1873 | /* @r{This is harmless. Simply move the unused} | |
1874 | @r{bytes to the beginning of the buffer so that} | |
1875 | @r{they can be used in the next round.} */ | |
1876 | memmove (inbuf, inptr, insize); | |
1877 | else | |
1878 | @{ | |
1879 | /* @r{It is a real problem. Maybe we ran out of} | |
1880 | @r{space in the output buffer or we have invalid} | |
1881 | @r{input. In any case back the file pointer to} | |
1882 | @r{the position of the last processed byte.} */ | |
1883 | lseek (fd, -insize, SEEK_CUR); | |
1884 | result = -1; | |
1885 | break; | |
1886 | @} | |
1887 | @} | |
1888 | @} | |
1889 | ||
1890 | /* @r{Terminate the output string.} */ | |
c559a3ca UD |
1891 | if (avail >= sizeof (wchar_t)) |
1892 | *((wchar_t *) wrptr) = L'\0'; | |
390955cb UD |
1893 | |
1894 | if (iconv_close (cd) != 0) | |
1895 | perror ("iconv_close"); | |
1896 | ||
1897 | return (wchar_t *) wrptr - outbuf; | |
1898 | @} | |
1899 | @end smallexample | |
1900 | ||
1901 | @cindex stateful | |
1902 | This example shows the most important aspects of using the @code{iconv} | |
1903 | functions. It shows how successive calls to @code{iconv} can be used to | |
1904 | convert large amounts of text. The user does not have to care about | |
1905 | stateful encodings as the functions take care of everything. | |
1906 | ||
1907 | An interesting point is the case where @code{iconv} return an error and | |
1908 | @code{errno} is set to @code{EINVAL}. This is not really an error in | |
1909 | the transformation. It can happen whenever the input character set | |
1910 | contains byte sequences of more than one byte for some character and | |
1911 | texts are not processed in one piece. In this case there is a chance | |
1912 | that a multibyte sequence is cut. The caller than can simply read the | |
1913 | remainder of the takes and feed the offending bytes together with new | |
1914 | character from the input to @code{iconv} and continue the work. The | |
1915 | internal state kept in the descriptor is @emph{not} unspecified after | |
1916 | such an event as it is the case with the conversion functions from the | |
1917 | @w{ISO C} standard. | |
1918 | ||
1919 | The example also shows the problem of using wide character strings with | |
1920 | @code{iconv}. As explained in the description of the @code{iconv} | |
1921 | function above the function always takes a pointer to a @code{char} | |
1922 | array and the available space is measured in bytes. In the example the | |
1923 | output buffer is a wide character buffer. Therefore we use a local | |
1924 | variable @var{wrptr} of type @code{char *} which is used in the | |
1925 | @code{iconv} calls. | |
1926 | ||
1927 | This looks rather innocent but can lead to problems on platforms which | |
1928 | have tight restriction on alignment. Therefore the caller of | |
1929 | @code{iconv} has to make sure that the pointers passed are suitable for | |
1930 | access of characters from the appropriate character set. Since in the | |
1931 | above case the input parameter to the function is a @code{wchar_t} | |
1932 | pointer this is the case (unless the user violates alignment when | |
1933 | computing the parameter). But in other situations, especially when | |
1934 | writing generic functions where one does not know what type of character | |
c1b2d472 | 1935 | set one uses and therefore treats text as a sequence of bytes, it might |
390955cb UD |
1936 | become tricky. |
1937 | ||
1938 | ||
1939 | @node Other iconv Implementations | |
1940 | @subsection Some Details about other @code{iconv} Implementations | |
1941 | ||
1942 | This is not really the place to discuss the @code{iconv} implementation | |
1943 | of other systems but it is necessary to know a bit about them to write | |
1944 | portable programs. The above mentioned problems with the specification | |
1945 | of the @code{iconv} functions can lead to portability issues. | |
1946 | ||
1947 | The first thing to notice is that due to the large number of character | |
1948 | sets in use it is certainly not practical to encode the conversions | |
1949 | directly in the C library. Therefore the conversion information must | |
d731df03 UD |
1950 | come from files outside the C library. This is usually done in one or |
1951 | both of the following ways: | |
390955cb UD |
1952 | |
1953 | @itemize @bullet | |
1954 | @item | |
1955 | The C library contains a set of generic conversion functions which can | |
1956 | read the needed conversion tables and other information from data files. | |
1957 | These files get loaded when necessary. | |
1958 | ||
d731df03 UD |
1959 | This solution is problematic as it requires a great deal of effort to |
1960 | apply to all character sets (potentially an infinite set). The | |
1961 | differences in the structure of the different character sets is so large | |
390955cb UD |
1962 | that many different variants of the table processing functions must be |
1963 | developed. On top of this the generic nature of these functions make | |
1964 | them slower than specifically implemented functions. | |
1965 | ||
1966 | @item | |
1967 | The C library only contains a framework which can dynamically load | |
1968 | object files and execute the therein contained conversion functions. | |
1969 | ||
1970 | This solution provides much more flexibility. The C library itself | |
1971 | contains only very little code and therefore reduces the general memory | |
1972 | footprint. Also, with a documented interface between the C library and | |
1973 | the loadable modules it is possible for third parties to extend the set | |
1974 | of available conversion modules. A drawback of this solution is that | |
1975 | dynamic loading must be available. | |
1976 | @end itemize | |
1977 | ||
1978 | Some implementations in commercial Unices implement a mixture of these | |
d731df03 UD |
1979 | these possibilities, the majority only the second solution. Using |
1980 | loadable modules moves the code out of the library itself and keeps the | |
1981 | door open for extensions and improvements. But this design is also | |
1982 | limiting on some platforms since not many platforms support dynamic | |
1983 | loading in statically linked programs. On platforms without his | |
1984 | capability it is therefore not possible to use this interface in | |
1985 | statically linked programs. The GNU C library has on ELF platforms no | |
1986 | problems with dynamic loading in in these situations and therefore this | |
b160422e | 1987 | point is moot. The danger is that one gets acquainted with this and |
d731df03 | 1988 | forgets about the restrictions on other systems. |
390955cb UD |
1989 | |
1990 | A second thing to know about other @code{iconv} implementations is that | |
c1b2d472 | 1991 | the number of available conversions is often very limited. Some |
d731df03 UD |
1992 | implementations provide in the standard release (not special |
1993 | international or developer releases) at most 100 to 200 conversion | |
1994 | possibilities. This does not mean 200 different character sets are | |
1995 | supported. E.g., conversions from one character set to a set of, say, | |
1996 | 10 others counts as 10 conversion. Together with the other direction | |
1997 | this makes already 20. One can imagine the thin coverage these platform | |
1998 | provide. Some Unix vendors even provide only a handful of conversions | |
1999 | which renders them useless for almost all uses. | |
390955cb UD |
2000 | |
2001 | This directly leads to a third and probably the most problematic point. | |
2002 | The way the @code{iconv} conversion functions are implemented on all | |
c1b2d472 | 2003 | known Unix system and the availability of the conversion functions from |
390955cb UD |
2004 | character set @math{@cal{A}} to @math{@cal{B}} and the conversion from |
2005 | @math{@cal{B}} to @math{@cal{C}} does @emph{not} imply that the | |
2006 | conversion from @math{@cal{A}} to @math{@cal{C}} is available. | |
2007 | ||
2008 | This might not seem unreasonable and problematic at first but it is a | |
2009 | quite big problem as one will notice shortly after hitting it. To show | |
2010 | the problem we assume to write a program which has to convert from | |
2011 | @math{@cal{A}} to @math{@cal{C}}. A call like | |
2012 | ||
2013 | @smallexample | |
2014 | cd = iconv_open ("@math{@cal{C}}", "@math{@cal{A}}"); | |
2015 | @end smallexample | |
2016 | ||
2017 | @noindent | |
2018 | does fail according to the assumption above. But what does the program | |
2019 | do now? The conversion is really necessary and therefore simply giving | |
2020 | up is no possibility. | |
2021 | ||
d731df03 UD |
2022 | This is a nuisance. The @code{iconv} function should take care of this. |
2023 | But how should the program proceed from here on? If it would try to | |
2024 | convert to character set @math{@cal{B}} first the two @code{iconv_open} | |
2025 | calls | |
390955cb UD |
2026 | |
2027 | @smallexample | |
2028 | cd1 = iconv_open ("@math{@cal{B}}", "@math{@cal{A}}"); | |
2029 | @end smallexample | |
2030 | ||
2031 | @noindent | |
2032 | and | |
2033 | ||
2034 | @smallexample | |
2035 | cd2 = iconv_open ("@math{@cal{C}}", "@math{@cal{B}}"); | |
2036 | @end smallexample | |
2037 | ||
2038 | @noindent | |
2039 | will succeed but how to find @math{@cal{B}}? | |
2040 | ||
d731df03 | 2041 | Unfortunately, the answer is: there is no general solution. On some |
390955cb | 2042 | systems guessing might help. On those systems most character sets can |
aaca11d8 | 2043 | convert to and from UTF-8 encoded @w{ISO 10646} or Unicode text. |
d731df03 | 2044 | Beside this only some very system-specific methods can help. Since the |
390955cb UD |
2045 | conversion functions come from loadable modules and these modules must |
2046 | be stored somewhere in the filesystem, one @emph{could} try to find them | |
2047 | and determine from the available file which conversions are available | |
2048 | and whether there is an indirect route from @math{@cal{A}} to | |
2049 | @math{@cal{C}}. | |
2050 | ||
2051 | This shows one of the design errors of @code{iconv} mentioned above. It | |
2052 | should at least be possible to determine the list of available | |
2053 | conversion programmatically so that if @code{iconv_open} says there is | |
2054 | no such conversion, one could make sure this also is true for indirect | |
2055 | routes. | |
2056 | ||
2057 | ||
2058 | @node glibc iconv Implementation | |
2059 | @subsection The @code{iconv} Implementation in the GNU C library | |
2060 | ||
2061 | After reading about the problems of @code{iconv} implementations in the | |
d731df03 UD |
2062 | last section it is certainly good to note that the implementation in |
2063 | the GNU C library has none of the problems mentioned above. What | |
2064 | follows is a step-by-step analysis of the points raised above. The | |
390955cb UD |
2065 | evaluation is based on the current state of the development (as of |
2066 | January 1999). The development of the @code{iconv} functions is not | |
49c091e5 | 2067 | complete, but basic functionality has solidified. |
390955cb UD |
2068 | |
2069 | The GNU C library's @code{iconv} implementation uses shared loadable | |
2070 | modules to implement the conversions. A very small number of | |
2071 | conversions are built into the library itself but these are only rather | |
2072 | trivial conversions. | |
2073 | ||
2074 | All the benefits of loadable modules are available in the GNU C library | |
d731df03 | 2075 | implementation. This is especially appealing since the interface is |
390955cb | 2076 | well documented (see below) and it therefore is easy to write new |
d731df03 | 2077 | conversion modules. The drawback of using loadable objects is not a |
390955cb UD |
2078 | problem in the GNU C library, at least on ELF systems. Since the |
2079 | library is able to load shared objects even in statically linked | |
d731df03 UD |
2080 | binaries this means that static linking needs not to be forbidden in |
2081 | case one wants to use @code{iconv}. | |
390955cb | 2082 | |
d731df03 UD |
2083 | The second mentioned problem is the number of supported conversions. |
2084 | Currently, the GNU C library supports more than 150 character sets. The | |
c1b2d472 UD |
2085 | way the implementation is designed the number of supported conversions |
2086 | is greater than 22350 (@math{150} times @math{149}). If any conversion | |
390955cb UD |
2087 | from or to a character set is missing it can easily be added. |
2088 | ||
d731df03 UD |
2089 | Particularly impressive as it may be, this high number is due to the |
2090 | fact that the GNU C library implementation of @code{iconv} does not have | |
2091 | the third problem mentioned above. I.e., whenever there is a conversion | |
2092 | from a character set @math{@cal{A}} to @math{@cal{B}} and from | |
2093 | @math{@cal{B}} to @math{@cal{C}} it is always possible to convert from | |
2094 | @math{@cal{A}} to @math{@cal{C}} directly. If the @code{iconv_open} | |
2095 | returns an error and sets @code{errno} to @code{EINVAL} this really | |
2096 | means there is no known way, directly or indirectly, to perform the | |
2097 | wanted conversion. | |
390955cb UD |
2098 | |
2099 | @cindex triangulation | |
2100 | This is achieved by providing for each character set a conversion from | |
aaca11d8 | 2101 | and to UCS-4 encoded @w{ISO 10646}. Using @w{ISO 10646} as an |
d731df03 UD |
2102 | intermediate representation it is possible to @dfn{triangulate}, i.e., |
2103 | converting with an intermediate representation. | |
390955cb UD |
2104 | |
2105 | There is no inherent requirement to provide a conversion to @w{ISO | |
2106 | 10646} for a new character set and it is also possible to provide other | |
d731df03 | 2107 | conversions where neither source nor destination character set is @w{ISO |
c1b2d472 | 2108 | 10646}. The currently existing set of conversions is simply meant to |
d731df03 | 2109 | cover all conversions which might be of interest. |
390955cb UD |
2110 | |
2111 | @cindex ISO-2022-JP | |
2112 | @cindex EUC-JP | |
d731df03 UD |
2113 | All currently available conversions use the triangulation method above, |
2114 | making conversion run unnecessarily slow. If, e.g., somebody often | |
2115 | needs the conversion from ISO-2022-JP to EUC-JP, a quicker solution | |
2116 | would involve direct conversion between the two character sets, skipping | |
2117 | the input to @w{ISO 10646} first. The two character sets of interest | |
2118 | are much more similar to each other than to @w{ISO 10646}. | |
390955cb UD |
2119 | |
2120 | In such a situation one can easy write a new conversion and provide it | |
2121 | as a better alternative. The GNU C library @code{iconv} implementation | |
2122 | would automatically use the module implementing the conversion if it is | |
2123 | specified to be more efficient. | |
2124 | ||
2125 | @subsubsection Format of @file{gconv-modules} files | |
2126 | ||
2127 | All information about the available conversions comes from a file named | |
2128 | @file{gconv-modules} which can be found in any of the directories along | |
2129 | the @code{GCONV_PATH}. The @file{gconv-modules} files are line-oriented | |
2130 | text files, where each of the lines has one of the following formats: | |
2131 | ||
2132 | @itemize @bullet | |
2133 | @item | |
2134 | If the first non-whitespace character is a @kbd{#} the line contains | |
c1b2d472 | 2135 | only comments and is ignored. |
390955cb UD |
2136 | |
2137 | @item | |
2138 | Lines starting with @code{alias} define an alias name for a character | |
2139 | set. There are two more words expected on the line. The first one | |
2140 | defines the alias name and the second defines the original name of the | |
2141 | character set. The effect is that it is possible to use the alias name | |
2142 | in the @var{fromset} or @var{toset} parameters of @code{iconv_open} and | |
2143 | achieve the same result as when using the real character set name. | |
2144 | ||
2145 | This is quite important as a character set has often many different | |
2146 | names. There is normally always an official name but this need not | |
2147 | correspond to the most popular name. Beside this many character sets | |
2148 | have special names which are somehow constructed. E.g., all character | |
2149 | sets specified by the ISO have an alias of the form | |
2150 | @code{ISO-IR-@var{nnn}} where @var{nnn} is the registration number. | |
2151 | This allows programs which know about the registration number to | |
2152 | construct character set names and use them in @code{iconv_open} calls. | |
c1b2d472 | 2153 | More on the available names and aliases follows below. |
390955cb UD |
2154 | |
2155 | @item | |
2156 | Lines starting with @code{module} introduce an available conversion | |
2157 | module. These lines must contain three or four more words. | |
2158 | ||
2159 | The first word specifies the source character set, the second word the | |
2160 | destination character set of conversion implemented in this module. The | |
2161 | third word is the name of the loadable module. The filename is | |
110215a9 | 2162 | constructed by appending the usual shared object suffix (normally |
390955cb UD |
2163 | @file{.so}) and this file is then supposed to be found in the same |
2164 | directory the @file{gconv-modules} file is in. The last word on the | |
2165 | line, which is optional, is a numeric value representing the cost of the | |
2166 | conversion. If this word is missing a cost of @math{1} is assumed. The | |
2167 | numeric value itself does not matter that much; what counts are the | |
2168 | relative values of the sums of costs for all possible conversion paths. | |
2169 | Below is a more precise description of the use of the cost value. | |
2170 | @end itemize | |
2171 | ||
d731df03 | 2172 | Returning to the example above where one has written a module to directly |
390955cb UD |
2173 | convert from ISO-2022-JP to EUC-JP and back. All what has to be done is |
2174 | to put the new module, be its name ISO2022JP-EUCJP.so, in a directory | |
2175 | and add a file @file{gconv-modules} with the following content in the | |
2176 | same directory: | |
2177 | ||
2178 | @smallexample | |
2179 | module ISO-2022-JP// EUC-JP// ISO2022JP-EUCJP 1 | |
2180 | module EUC-JP// ISO-2022-JP// ISO2022JP-EUCJP 1 | |
2181 | @end smallexample | |
2182 | ||
d731df03 UD |
2183 | To see why this is sufficient, it is necessary to understand how the |
2184 | conversion used by @code{iconv} (and described in the descriptor) is | |
390955cb UD |
2185 | selected. The approach to this problem is quite simple. |
2186 | ||
2187 | At the first call of the @code{iconv_open} function the program reads | |
2188 | all available @file{gconv-modules} files and builds up two tables: one | |
2189 | containing all the known aliases and another which contains the | |
2190 | information about the conversions and which shared object implements | |
2191 | them. | |
2192 | ||
2193 | @subsubsection Finding the conversion path in @code{iconv} | |
2194 | ||
2195 | The set of available conversions form a directed graph with weighted | |
d731df03 UD |
2196 | edges. The weights on the edges are the costs specified in the |
2197 | @file{gconv-modules} files. The @code{iconv_open} function uses an | |
2198 | algorithm suitable for search for the best path in such a graph and so | |
2199 | constructs a list of conversions which must be performed in succession | |
2200 | to get the transformation from the source to the destination character | |
2201 | set. | |
2202 | ||
2203 | Explaining why the above @file{gconv-modules} files allows the | |
2204 | @code{iconv} implementation to resolve the specific ISO-2022-JP to | |
2205 | EUC-JP conversion module instead of the conversion coming with the | |
49c091e5 | 2206 | library itself is straightforward. Since the latter conversion takes two |
d731df03 | 2207 | steps (from ISO-2022-JP to @w{ISO 10646} and then from @w{ISO 10646} to |
390955cb UD |
2208 | EUC-JP) the cost is @math{1+1 = 2}. But the above @file{gconv-modules} |
2209 | file specifies that the new conversion modules can perform this | |
2210 | conversion with only the cost of @math{1}. | |
2211 | ||
d731df03 UD |
2212 | A mysterious piece about the @file{gconv-modules} file above (and also |
2213 | the file coming with the GNU C library) are the names of the character | |
2214 | sets specified in the @code{module} lines. Why do almost all the names | |
2215 | end in @code{//}? And this is not all: the names can actually be | |
2216 | regular expressions. At this point of time this mystery should not be | |
2217 | revealed, unless you have the relevant spell-casting materials: ashes | |
2218 | from an original @w{DOS 6.2} boot disk burnt in effigy, a crucifix | |
2219 | blessed by St.@: Emacs, assorted herbal roots from Central America, sand | |
2220 | from Cebu, etc. Sorry! @strong{The part of the implementation where | |
2221 | this is used is not yet finished. For now please simply follow the | |
2222 | existing examples. It'll become clearer once it is. --drepper} | |
390955cb UD |
2223 | |
2224 | A last remark about the @file{gconv-modules} is about the names not | |
2225 | ending with @code{//}. There often is a character set named | |
2226 | @code{INTERNAL} mentioned. From the discussion above and the chosen | |
110215a9 | 2227 | name it should have become clear that this is the name for the |
390955cb | 2228 | representation used in the intermediate step of the triangulation. We |
aaca11d8 UD |
2229 | have said that this is UCS-4 but actually it is not quite right. The |
2230 | UCS-4 specification also includes the specification of the byte ordering | |
2231 | used. Since a UCS-4 value consists of four bytes a stored value is | |
390955cb | 2232 | effected by byte ordering. The internal representation is @emph{not} |
aaca11d8 UD |
2233 | the same as UCS-4 in case the byte ordering of the processor (or at least |
2234 | the running process) is not the same as the one required for UCS-4. This | |
390955cb UD |
2235 | is done for performance reasons as one does not want to perform |
2236 | unnecessary byte-swapping operations if one is not interested in actually | |
aaca11d8 | 2237 | seeing the result in UCS-4. To avoid trouble with endianess the internal |
390955cb UD |
2238 | representation consistently is named @code{INTERNAL} even on big-endian |
2239 | systems where the representations are identical. | |
2240 | ||
2241 | @subsubsection @code{iconv} module data structures | |
2242 | ||
2243 | So far this section described how modules are located and considered to | |
2244 | be used. What remains to be described is the interface of the modules | |
2245 | so that one can write new ones. This section describes the interface as | |
2246 | it is in use in January 1999. The interface will change in future a bit | |
2247 | but hopefully only in an upward compatible way. | |
2248 | ||
49c091e5 | 2249 | The definitions necessary to write new modules are publicly available |
390955cb UD |
2250 | in the non-standard header @file{gconv.h}. The following text will |
2251 | therefore describe the definitions from this header file. But first it | |
2252 | is necessary to get an overview. | |
2253 | ||
2254 | From the perspective of the user of @code{iconv} the interface is quite | |
2255 | simple: the @code{iconv_open} function returns a handle which can be | |
110215a9 | 2256 | used in calls to @code{iconv} and finally the handle is freed with a call |
390955cb UD |
2257 | to @code{iconv_close}. The problem is: the handle has to be able to |
2258 | represent the possibly long sequences of conversion steps and also the | |
2259 | state of each conversion since the handle is all which is passed to the | |
2260 | @code{iconv} function. Therefore the data structures are really the | |
2261 | elements to understanding the implementation. | |
2262 | ||
2263 | We need two different kinds of data structures. The first describes the | |
2264 | conversion and the second describes the state etc. There are really two | |
2265 | type definitions like this in @file{gconv.h}. | |
2266 | @pindex gconv.h | |
2267 | ||
2268 | @comment gconv.h | |
2269 | @comment GNU | |
976780fd | 2270 | @deftp {Data type} {struct __gconv_step} |
390955cb UD |
2271 | This data structure describes one conversion a module can perform. For |
2272 | each function in a loaded module with conversion functions there is | |
2273 | exactly one object of this type. This object is shared by all users of | |
2274 | the conversion. I.e., this object does not contain any information | |
2275 | corresponding to an actual conversion. It only describes the conversion | |
2276 | itself. | |
2277 | ||
2278 | @table @code | |
976780fd UD |
2279 | @item struct __gconv_loaded_object *__shlib_handle |
2280 | @itemx const char *__modname | |
2281 | @itemx int __counter | |
390955cb UD |
2282 | All these elements of the structure are used internally in the C library |
2283 | to coordinate loading and unloading the shared. One must not expect any | |
2284 | of the other elements be available or initialized. | |
2285 | ||
976780fd UD |
2286 | @item const char *__from_name |
2287 | @itemx const char *__to_name | |
2288 | @code{__from_name} and @code{__to_name} contain the names of the source and | |
390955cb UD |
2289 | destination character sets. They can be used to identify the actual |
2290 | conversion to be carried out since one module might implement | |
2291 | conversions for more than one character set and/or direction. | |
2292 | ||
976780fd UD |
2293 | @item gconv_fct __fct |
2294 | @itemx gconv_init_fct __init_fct | |
2295 | @itemx gconv_end_fct __end_fct | |
390955cb UD |
2296 | These elements contain pointers to the functions in the loadable module. |
2297 | The interface will be explained below. | |
2298 | ||
976780fd UD |
2299 | @item int __min_needed_from |
2300 | @itemx int __max_needed_from | |
2301 | @itemx int __min_needed_to | |
2302 | @itemx int __max_needed_to; | |
2303 | These values have to be filled in the init function of the module. The | |
2304 | @code{__min_needed_from} value specifies how many bytes a character of | |
2305 | the source character set at least needs. The @code{__max_needed_from} | |
390955cb UD |
2306 | specifies the maximum value which also includes possible shift |
2307 | sequences. | |
2308 | ||
976780fd UD |
2309 | The @code{__min_needed_to} and @code{__max_needed_to} values serve the |
2310 | same purpose but this time for the destination character set. | |
390955cb UD |
2311 | |
2312 | It is crucial that these values are accurate since otherwise the | |
2313 | conversion functions will have problems or not work at all. | |
2314 | ||
976780fd | 2315 | @item int __stateful |
390955cb UD |
2316 | This element must also be initialized by the init function. It is |
2317 | nonzero if the source character set is stateful. Otherwise it is zero. | |
2318 | ||
976780fd | 2319 | @item void *__data |
390955cb UD |
2320 | This element can be used freely by the conversion functions in the |
2321 | module. It can be used to communicate extra information from one call | |
2322 | to another. It need not be initialized if not needed at all. If this | |
2323 | element gets assigned a pointer to dynamically allocated memory | |
2324 | (presumably in the init function) it has to be made sure that the end | |
2325 | function deallocates the memory. Otherwise the application will leak | |
2326 | memory. | |
2327 | ||
2328 | It is important to be aware that this data structure is shared by all | |
976780fd | 2329 | users of this specification conversion and therefore the @code{__data} |
390955cb UD |
2330 | element must not contain data specific to one specific use of the |
2331 | conversion function. | |
2332 | @end table | |
2333 | @end deftp | |
2334 | ||
2335 | @comment gconv.h | |
2336 | @comment GNU | |
976780fd | 2337 | @deftp {Data type} {struct __gconv_step_data} |
390955cb UD |
2338 | This is the data structure which contains the information specific to |
2339 | each use of the conversion functions. | |
2340 | ||
2341 | @table @code | |
976780fd UD |
2342 | @item char *__outbuf |
2343 | @itemx char *__outbufend | |
390955cb | 2344 | These elements specify the output buffer for the conversion step. The |
976780fd UD |
2345 | @code{__outbuf} element points to the beginning of the buffer and |
2346 | @code{__outbufend} points to the byte following the last byte in the | |
390955cb UD |
2347 | buffer. The conversion function must not assume anything about the size |
2348 | of the buffer but it can be safely assumed the there is room for at | |
2349 | least one complete character in the output buffer. | |
2350 | ||
2351 | Once the conversion is finished and the conversion is the last step the | |
976780fd | 2352 | @code{__outbuf} element must be modified to point after last last byte |
390955cb UD |
2353 | written into the buffer to signal how much output is available. If this |
2354 | conversion step is not the last one the element must not be modified. | |
976780fd | 2355 | The @code{__outbufend} element must not be modified. |
390955cb | 2356 | |
976780fd | 2357 | @item int __is_last |
390955cb UD |
2358 | This element is nonzero if this conversion step is the last one. This |
2359 | information is necessary for the recursion. See the description of the | |
2360 | conversion function internals below. This element must never be | |
2361 | modified. | |
2362 | ||
976780fd | 2363 | @item int __invocation_counter |
390955cb UD |
2364 | The conversion function can use this element to see how many calls of |
2365 | the conversion function already happened. Some character sets require | |
2366 | when generating output a certain prolog and by comparing this value with | |
2367 | zero one can find out whether it is the first call and therefore the | |
2368 | prolog should be emitted or not. This element must never be modified. | |
2369 | ||
976780fd | 2370 | @item int __internal_use |
390955cb UD |
2371 | This element is another one rarely used but needed in certain |
2372 | situations. It got assigned a nonzero value in case the conversion | |
2373 | functions are used to implement @code{mbsrtowcs} et.al. I.e., the | |
2374 | function is not used directly through the @code{iconv} interface. | |
2375 | ||
2376 | This sometimes makes a difference as it is expected that the | |
2377 | @code{iconv} functions are used to translate entire texts while the | |
2378 | @code{mbsrtowcs} functions are normally only used to convert single | |
2379 | strings and might be used multiple times to convert entire texts. | |
2380 | ||
2381 | But in this situation we would have problem complying with some rules of | |
2382 | the character set specification. Some character sets require a prolog | |
2383 | which must appear exactly once for an entire text. If a number of | |
2384 | @code{mbsrtowcs} calls are used to convert the text only the first call | |
2385 | must add the prolog. But since there is no communication between the | |
2386 | different calls of @code{mbsrtowcs} the conversion functions have no | |
2387 | possibility to find this out. The situation is different for sequences | |
e8b1163e | 2388 | of @code{iconv} calls since the handle allows access to the needed |
390955cb UD |
2389 | information. |
2390 | ||
976780fd UD |
2391 | This element is mostly used together with @code{__invocation_counter} in |
2392 | a way like this: | |
390955cb UD |
2393 | |
2394 | @smallexample | |
976780fd UD |
2395 | if (!data->__internal_use |
2396 | && data->__invocation_counter == 0) | |
390955cb UD |
2397 | /* @r{Emit prolog.} */ |
2398 | ... | |
2399 | @end smallexample | |
2400 | ||
2401 | This element must never be modified. | |
2402 | ||
976780fd UD |
2403 | @item mbstate_t *__statep |
2404 | The @code{__statep} element points to an object of type @code{mbstate_t} | |
c1b2d472 | 2405 | (@pxref{Keeping the state}). The conversion of an stateful character |
390955cb | 2406 | set must use the object pointed to by this element to store information |
976780fd UD |
2407 | about the conversion state. The @code{__statep} element itself must |
2408 | never be modified. | |
390955cb UD |
2409 | |
2410 | @item mbstate_t __state | |
2411 | This element @emph{never} must be used directly. It is only part of | |
2412 | this structure to have the needed space allocated. | |
2413 | @end table | |
2414 | @end deftp | |
2415 | ||
2416 | @subsubsection @code{iconv} module interfaces | |
2417 | ||
2418 | With the knowledge about the data structures we now can describe the | |
2419 | conversion functions itself. To understand the interface a bit of | |
2420 | knowledge about the functionality in the C library which loads the | |
2421 | objects with the conversions is necessary. | |
2422 | ||
2423 | It is often the case that one conversion is used more than once. I.e., | |
2424 | there are several @code{iconv_open} calls for the same set of character | |
2425 | sets during one program run. The @code{mbsrtowcs} et.al.@: functions in | |
2426 | the GNU C library also use the @code{iconv} functionality which | |
2427 | increases the number of uses of the same functions even more. | |
2428 | ||
2429 | For this reason the modules do not get loaded exclusively for one | |
49c091e5 | 2430 | conversion. Instead a module once loaded can be used by arbitrarily many |
390955cb UD |
2431 | @code{iconv} or @code{mbsrtowcs} calls at the same time. The splitting |
2432 | of the information between conversion function specific information and | |
2433 | conversion data makes this possible. The last section showed the two | |
49c091e5 | 2434 | data structures used to do this. |
390955cb | 2435 | |
49c091e5 | 2436 | This is of course also reflected in the interface and semantics of the |
390955cb UD |
2437 | functions the modules must provide. There are three functions which |
2438 | must have the following names: | |
2439 | ||
2440 | @table @code | |
2441 | @item gconv_init | |
2442 | The @code{gconv_init} function initializes the conversion function | |
2443 | specific data structure. This very same object is shared by all | |
2444 | conversion which use this conversion and therefore no state information | |
2445 | about the conversion itself must be stored in here. If a module | |
2446 | implements more than one conversion the @code{gconv_init} function will be | |
2447 | called multiple times. | |
2448 | ||
2449 | @item gconv_end | |
2450 | The @code{gconv_end} function is responsible to free all resources | |
2451 | allocated by the @code{gconv_init} function. If there is nothing to do | |
2452 | this function can be missing. Special care must be taken if the module | |
2453 | implements more than one conversion and the @code{gconv_init} function | |
2454 | does not allocate the same resources for all conversions. | |
2455 | ||
2456 | @item gconv | |
2457 | This is the actual conversion function. It is called to convert one | |
2458 | block of text. It gets passed the conversion step information | |
2459 | initialized by @code{gconv_init} and the conversion data, specific to | |
2460 | this use of the conversion functions. | |
2461 | @end table | |
2462 | ||
2463 | There are three data types defined for the three module interface | |
2464 | function and these define the interface. | |
2465 | ||
2466 | @comment gconv.h | |
2467 | @comment GNU | |
fef0b717 | 2468 | @deftypevr {Data type} int {(*__gconv_init_fct)} (struct __gconv_step *) |
390955cb UD |
2469 | This specifies the interface of the initialization function of the |
2470 | module. It is called exactly once for each conversion the module | |
2471 | implements. | |
2472 | ||
976780fd | 2473 | As explained int the description of the @code{struct __gconv_step} data |
390955cb UD |
2474 | structure above the initialization function has to initialize parts of |
2475 | it. | |
2476 | ||
2477 | @table @code | |
976780fd UD |
2478 | @item __min_needed_from |
2479 | @itemx __max_needed_from | |
2480 | @itemx __min_needed_to | |
2481 | @itemx __max_needed_to | |
390955cb UD |
2482 | These elements must be initialized to the exact numbers of the minimum |
2483 | and maximum number of bytes used by one character in the source and | |
2484 | destination character set respectively. If the characters all have the | |
2485 | same size the minimum and maximum values are the same. | |
2486 | ||
976780fd | 2487 | @item __stateful |
390955cb UD |
2488 | This element must be initialized to an nonzero value if the source |
2489 | character set is stateful. Otherwise it must be zero. | |
2490 | @end table | |
2491 | ||
2492 | If the initialization function needs to communication some information | |
976780fd UD |
2493 | to the conversion function this can happen using the @code{__data} |
2494 | element of the @code{__gconv_step} structure. But since this data is | |
2495 | shared by all the conversion is must not be modified by the conversion | |
2496 | function. How this can be used is shown in the example below. | |
390955cb UD |
2497 | |
2498 | @smallexample | |
2499 | #define MIN_NEEDED_FROM 1 | |
2500 | #define MAX_NEEDED_FROM 4 | |
2501 | #define MIN_NEEDED_TO 4 | |
2502 | #define MAX_NEEDED_TO 4 | |
2503 | ||
2504 | int | |
976780fd | 2505 | gconv_init (struct __gconv_step *step) |
390955cb UD |
2506 | @{ |
2507 | /* @r{Determine which direction.} */ | |
2508 | struct iso2022jp_data *new_data; | |
2509 | enum direction dir = illegal_dir; | |
2510 | enum variant var = illegal_var; | |
2511 | int result; | |
2512 | ||
976780fd | 2513 | if (__strcasecmp (step->__from_name, "ISO-2022-JP//") == 0) |
390955cb UD |
2514 | @{ |
2515 | dir = from_iso2022jp; | |
2516 | var = iso2022jp; | |
2517 | @} | |
976780fd | 2518 | else if (__strcasecmp (step->__to_name, "ISO-2022-JP//") == 0) |
390955cb UD |
2519 | @{ |
2520 | dir = to_iso2022jp; | |
2521 | var = iso2022jp; | |
2522 | @} | |
976780fd | 2523 | else if (__strcasecmp (step->__from_name, "ISO-2022-JP-2//") == 0) |
390955cb UD |
2524 | @{ |
2525 | dir = from_iso2022jp; | |
2526 | var = iso2022jp2; | |
2527 | @} | |
976780fd | 2528 | else if (__strcasecmp (step->__to_name, "ISO-2022-JP-2//") == 0) |
390955cb UD |
2529 | @{ |
2530 | dir = to_iso2022jp; | |
2531 | var = iso2022jp2; | |
2532 | @} | |
2533 | ||
976780fd | 2534 | result = __GCONV_NOCONV; |
390955cb UD |
2535 | if (dir != illegal_dir) |
2536 | @{ | |
2537 | new_data = (struct iso2022jp_data *) | |
2538 | malloc (sizeof (struct iso2022jp_data)); | |
2539 | ||
976780fd | 2540 | result = __GCONV_NOMEM; |
390955cb UD |
2541 | if (new_data != NULL) |
2542 | @{ | |
2543 | new_data->dir = dir; | |
2544 | new_data->var = var; | |
976780fd | 2545 | step->__data = new_data; |
390955cb UD |
2546 | |
2547 | if (dir == from_iso2022jp) | |
2548 | @{ | |
976780fd UD |
2549 | step->__min_needed_from = MIN_NEEDED_FROM; |
2550 | step->__max_needed_from = MAX_NEEDED_FROM; | |
2551 | step->__min_needed_to = MIN_NEEDED_TO; | |
2552 | step->__max_needed_to = MAX_NEEDED_TO; | |
390955cb UD |
2553 | @} |
2554 | else | |
2555 | @{ | |
976780fd UD |
2556 | step->__min_needed_from = MIN_NEEDED_TO; |
2557 | step->__max_needed_from = MAX_NEEDED_TO; | |
2558 | step->__min_needed_to = MIN_NEEDED_FROM; | |
2559 | step->__max_needed_to = MAX_NEEDED_FROM + 2; | |
390955cb UD |
2560 | @} |
2561 | ||
2562 | /* @r{Yes, this is a stateful encoding.} */ | |
976780fd | 2563 | step->__stateful = 1; |
390955cb | 2564 | |
976780fd | 2565 | result = __GCONV_OK; |
390955cb UD |
2566 | @} |
2567 | @} | |
2568 | ||
2569 | return result; | |
2570 | @} | |
2571 | @end smallexample | |
2572 | ||
2573 | The function first checks which conversion is wanted. The module from | |
2574 | which this function is taken implements four different conversion and | |
2575 | which one is selected can be determined by comparing the names. The | |
2576 | comparison should always be done without paying attention to the case. | |
2577 | ||
2578 | Then a data structure is allocated which contains the necessary | |
2579 | information about which conversion is selected. The data structure | |
2580 | @code{struct iso2022jp_data} is locally defined since outside the module | |
2581 | this data is not used at all. Please note that if all four conversions | |
2582 | this modules supports are requested there are four data blocks. | |
2583 | ||
976780fd UD |
2584 | One interesting thing is the initialization of the @code{__min_} and |
2585 | @code{__max_} elements of the step data object. A single ISO-2022-JP | |
390955cb UD |
2586 | character can consist of one to four bytes. Therefore the |
2587 | @code{MIN_NEEDED_FROM} and @code{MAX_NEEDED_FROM} macros are defined | |
2588 | this way. The output is always the @code{INTERNAL} character set (aka | |
aaca11d8 | 2589 | UCS-4) and therefore each character consists of exactly four bytes. For |
390955cb UD |
2590 | the conversion from @code{INTERNAL} to ISO-2022-JP we have to take into |
2591 | account that escape sequences might be necessary to switch the character | |
976780fd | 2592 | sets. Therefore the @code{__max_needed_to} element for this direction |
390955cb UD |
2593 | gets assigned @code{MAX_NEEDED_FROM + 2}. This takes into account the |
2594 | two bytes needed for the escape sequences to single the switching. The | |
2595 | asymmetry in the maximum values for the two directions can be explained | |
2596 | easily: when reading ISO-2022-JP text escape sequences can be handled | |
2597 | alone. I.e., it is not necessary to process a real character since the | |
2598 | effect of the escape sequence can be recorded in the state information. | |
2599 | The situation is different for the other direction. Since it is in | |
2600 | general not known which character comes next one cannot emit escape | |
2601 | sequences to change the state in advance. This means the escape | |
2602 | sequences which have to be emitted together with the next character. | |
2603 | Therefore one needs more room then only for the character itself. | |
2604 | ||
2605 | The possible return values of the initialization function are: | |
2606 | ||
2607 | @table @code | |
976780fd | 2608 | @item __GCONV_OK |
390955cb | 2609 | The initialization succeeded |
976780fd | 2610 | @item __GCONV_NOCONV |
390955cb UD |
2611 | The requested conversion is not supported in the module. This can |
2612 | happen if the @file{gconv-modules} file has errors. | |
976780fd | 2613 | @item __GCONV_NOMEM |
390955cb UD |
2614 | Memory required to store additional information could not be allocated. |
2615 | @end table | |
2616 | @end deftypevr | |
2617 | ||
2618 | The functions called before the module is unloaded is significantly | |
2619 | easier. It often has nothing at all to do in which case it can be left | |
2620 | out completely. | |
2621 | ||
2622 | @comment gconv.h | |
2623 | @comment GNU | |
fef0b717 | 2624 | @deftypevr {Data type} void {(*__gconv_end_fct)} (struct gconv_step *) |
390955cb | 2625 | The task of this function is it to free all resources allocated in the |
976780fd UD |
2626 | initialization function. Therefore only the @code{__data} element of |
2627 | the object pointed to by the argument is of interest. Continuing the | |
390955cb UD |
2628 | example from the initialization function, the finalization function |
2629 | looks like this: | |
2630 | ||
2631 | @smallexample | |
2632 | void | |
976780fd | 2633 | gconv_end (struct __gconv_step *data) |
390955cb | 2634 | @{ |
976780fd | 2635 | free (data->__data); |
390955cb UD |
2636 | @} |
2637 | @end smallexample | |
2638 | @end deftypevr | |
2639 | ||
d731df03 UD |
2640 | The most important function is the conversion function itself. It can |
2641 | get quite complicated for complex character sets. But since this is not | |
2642 | of interest here we will only describe a possible skeleton for the | |
2643 | conversion function. | |
390955cb UD |
2644 | |
2645 | @comment gconv.h | |
2646 | @comment GNU | |
fef0b717 | 2647 | @deftypevr {Data type} int {(*__gconv_fct)} (struct __gconv_step *, struct __gconv_step_data *, const char **, const char *, size_t *, int) |
390955cb UD |
2648 | The conversion function can be called for two basic reason: to convert |
2649 | text or to reset the state. From the description of the @code{iconv} | |
2650 | function it can be seen why the flushing mode is necessary. What mode | |
2651 | is selected is determined by the sixth argument, an integer. If it is | |
2652 | nonzero it means that flushing is selected. | |
2653 | ||
2654 | Common to both mode is where the output buffer can be found. The | |
2655 | information about this buffer is stored in the conversion step data. A | |
2656 | pointer to this is passed as the second argument to this function. The | |
976780fd | 2657 | description of the @code{struct __gconv_step_data} structure has more |
390955cb UD |
2658 | information on this. |
2659 | ||
2660 | @cindex stateful | |
2661 | What has to be done for flushing depends on the source character set. | |
2662 | If it is not stateful nothing has to be done. Otherwise the function | |
2663 | has to emit a byte sequence to bring the state object in the initial | |
2664 | state. Once this all happened the other conversion modules in the chain | |
2665 | of conversions have to get the same chance. Whether another step | |
976780fd | 2666 | follows can be determined from the @code{__is_last} element of the step |
390955cb UD |
2667 | data structure to which the first parameter points. |
2668 | ||
2669 | The more interesting mode is when actually text has to be converted. | |
2670 | The first step in this case is to convert as much text as possible from | |
2671 | the input buffer and store the result in the output buffer. The start | |
2672 | of the input buffer is determined by the third argument which is a | |
2673 | pointer to a pointer variable referencing the beginning of the buffer. | |
2674 | The fourth argument is a pointer to the byte right after the last byte | |
2675 | in the buffer. | |
2676 | ||
2677 | The conversion has to be performed according to the current state if the | |
2678 | character set is stateful. The state is stored in an object pointed to | |
976780fd | 2679 | by the @code{__statep} element of the step data (second argument). Once |
390955cb UD |
2680 | either the input buffer is empty or the output buffer is full the |
2681 | conversion stops. At this point the pointer variable referenced by the | |
2682 | third parameter must point to the byte following the last processed | |
2683 | byte. I.e., if all of the input is consumed this pointer and the fourth | |
2684 | parameter have the same value. | |
2685 | ||
2686 | What now happens depends on whether this step is the last one or not. | |
2687 | If it is the last step the only thing which has to be done is to update | |
976780fd | 2688 | the @code{__outbuf} element of the step data structure to point after the |
390955cb UD |
2689 | last written byte. This gives the caller the information on how much |
2690 | text is available in the output buffer. Beside this the variable | |
2691 | pointed to by the fifth parameter, which is of type @code{size_t}, must | |
2692 | be incremented by the number of characters (@emph{not bytes}) which were | |
e3a7ff87 | 2693 | converted in a non-reversible way. Then the function can return. |
390955cb UD |
2694 | |
2695 | In case the step is not the last one the later conversion functions have | |
2696 | to get a chance to do their work. Therefore the appropriate conversion | |
2697 | function has to be called. The information about the functions is | |
2698 | stored in the conversion data structures, passed as the first parameter. | |
2699 | This information and the step data are stored in arrays so the next | |
2700 | element in both cases can be found by simple pointer arithmetic: | |
2701 | ||
2702 | @smallexample | |
2703 | int | |
976780fd | 2704 | gconv (struct __gconv_step *step, struct __gconv_step_data *data, |
390955cb UD |
2705 | const char **inbuf, const char *inbufend, size_t *written, |
2706 | int do_flush) | |
2707 | @{ | |
976780fd UD |
2708 | struct __gconv_step *next_step = step + 1; |
2709 | struct __gconv_step_data *next_data = data + 1; | |
390955cb UD |
2710 | ... |
2711 | @end smallexample | |
2712 | ||
2713 | The @code{next_step} pointer references the next step information and | |
2714 | @code{next_data} the next data record. The call of the next function | |
2715 | therefore will look similar to this: | |
2716 | ||
2717 | @smallexample | |
976780fd UD |
2718 | next_step->__fct (next_step, next_data, &outerr, outbuf, |
2719 | written, 0) | |
390955cb UD |
2720 | @end smallexample |
2721 | ||
2722 | But this is not yet all. Once the function call returns the conversion | |
2723 | function might have some more to do. If the return value of the | |
976780fd | 2724 | function is @code{__GCONV_EMPTY_INPUT} this means there is more room in |
390955cb UD |
2725 | the output buffer. Unless the input buffer is empty the conversion |
2726 | functions start all over again and processes the rest of the input | |
976780fd | 2727 | buffer. If the return value is not @code{__GCONV_EMPTY_INPUT} something |
390955cb UD |
2728 | went wrong and we have to recover from this. |
2729 | ||
2730 | A requirement for the conversion function is that the input buffer | |
2731 | pointer (the third argument) always points to the last character which | |
e3a7ff87 | 2732 | was put in the converted form in the output buffer. This is trivially |
390955cb UD |
2733 | true after the conversion performed in the current step. But if the |
2734 | conversion functions deeper down the stream stop prematurely not all | |
2735 | characters from the output buffer are consumed and therefore the input | |
2736 | buffer pointers must be backed of to the right position. | |
2737 | ||
2738 | This is easy to do if the input and output character sets have a fixed | |
2739 | width for all characters. In this situation we can compute how many | |
2740 | characters are left in the output buffer and therefore can correct the | |
2741 | input buffer pointer appropriate with a similar computation. Things are | |
2742 | getting tricky if either character set has character represented with | |
2743 | variable length byte sequences and it gets even more complicated if the | |
2744 | conversion has to take care of the state. In these cases the conversion | |
2745 | has to be performed once again, from the known state before the initial | |
2746 | conversion. I.e., if necessary the state of the conversion has to be | |
2747 | reset and the conversion loop has to be executed again. The difference | |
2748 | now is that it is known how much input must be created and the | |
2749 | conversion can stop before converting the first unused character. Once | |
2750 | this is done the input buffer pointers must be updated again and the | |
2751 | function can return. | |
2752 | ||
2753 | One final thing should be mentioned. If it is necessary for the | |
2754 | conversion to know whether it is the first invocation (in case a prolog | |
2755 | has to be emitted) the conversion function should just before returning | |
976780fd | 2756 | to the caller increment the @code{__invocation_counter} element of the |
390955cb | 2757 | step data structure. See the description of the @code{struct |
976780fd UD |
2758 | __gconv_step_data} structure above for more information on how this can |
2759 | be used. | |
390955cb UD |
2760 | |
2761 | The return value must be one of the following values: | |
2762 | ||
2763 | @table @code | |
976780fd | 2764 | @item __GCONV_EMPTY_INPUT |
390955cb | 2765 | All input was consumed and there is room left in the output buffer. |
1410e233 | 2766 | @item __GCONV_FULL_OUTPUT |
390955cb UD |
2767 | No more room in the output buffer. In case this is not the last step |
2768 | this value is propagated down from the call of the next conversion | |
2769 | function in the chain. | |
976780fd | 2770 | @item __GCONV_INCOMPLETE_INPUT |
390955cb UD |
2771 | The input buffer is not entirely empty since it contains an incomplete |
2772 | character sequence. | |
2773 | @end table | |
2774 | ||
2775 | The following example provides a framework for a conversion function. | |
2776 | In case a new conversion has to be written the holes in this | |
2777 | implementation have to be filled and that is it. | |
2778 | ||
2779 | @smallexample | |
2780 | int | |
976780fd | 2781 | gconv (struct __gconv_step *step, struct __gconv_step_data *data, |
390955cb UD |
2782 | const char **inbuf, const char *inbufend, size_t *written, |
2783 | int do_flush) | |
2784 | @{ | |
976780fd UD |
2785 | struct __gconv_step *next_step = step + 1; |
2786 | struct __gconv_step_data *next_data = data + 1; | |
2787 | gconv_fct fct = next_step->__fct; | |
390955cb UD |
2788 | int status; |
2789 | ||
2790 | /* @r{If the function is called with no input this means we have} | |
2791 | @r{to reset to the initial state. The possibly partly} | |
2792 | @r{converted input is dropped.} */ | |
2793 | if (do_flush) | |
2794 | @{ | |
976780fd | 2795 | status = __GCONV_OK; |
390955cb UD |
2796 | |
2797 | /* @r{Possible emit a byte sequence which put the state object} | |
2798 | @r{into the initial state.} */ | |
2799 | ||
2800 | /* @r{Call the steps down the chain if there are any but only} | |
2801 | @r{if we successfully emitted the escape sequence.} */ | |
976780fd | 2802 | if (status == __GCONV_OK && ! data->__is_last) |
390955cb UD |
2803 | status = fct (next_step, next_data, NULL, NULL, |
2804 | written, 1); | |
2805 | @} | |
2806 | else | |
2807 | @{ | |
2808 | /* @r{We preserve the initial values of the pointer variables.} */ | |
2809 | const char *inptr = *inbuf; | |
976780fd UD |
2810 | char *outbuf = data->__outbuf; |
2811 | char *outend = data->__outbufend; | |
390955cb UD |
2812 | char *outptr; |
2813 | ||
390955cb UD |
2814 | do |
2815 | @{ | |
2816 | /* @r{Remember the start value for this round.} */ | |
2817 | inptr = *inbuf; | |
2818 | /* @r{The outbuf buffer is empty.} */ | |
2819 | outptr = outbuf; | |
2820 | ||
2821 | /* @r{For stateful encodings the state must be safe here.} */ | |
2822 | ||
2823 | /* @r{Run the conversion loop. @code{status} is set} | |
2824 | @r{appropriately afterwards.} */ | |
2825 | ||
2826 | /* @r{If this is the last step leave the loop, there is} | |
2827 | @r{nothing we can do.} */ | |
976780fd | 2828 | if (data->__is_last) |
390955cb UD |
2829 | @{ |
2830 | /* @r{Store information about how many bytes are} | |
2831 | @r{available.} */ | |
976780fd | 2832 | data->__outbuf = outbuf; |
390955cb | 2833 | |
e3a7ff87 UD |
2834 | /* @r{If any non-reversible conversions were performed,} |
2835 | @r{add the number to @code{*written}.} */ | |
390955cb UD |
2836 | |
2837 | break; | |
2838 | @} | |
2839 | ||
2840 | /* @r{Write out all output which was produced.} */ | |
2841 | if (outbuf > outptr) | |
2842 | @{ | |
976780fd | 2843 | const char *outerr = data->__outbuf; |
390955cb UD |
2844 | int result; |
2845 | ||
2846 | result = fct (next_step, next_data, &outerr, | |
2847 | outbuf, written, 0); | |
2848 | ||
976780fd | 2849 | if (result != __GCONV_EMPTY_INPUT) |
390955cb UD |
2850 | @{ |
2851 | if (outerr != outbuf) | |
2852 | @{ | |
2853 | /* @r{Reset the input buffer pointer. We} | |
2854 | @r{document here the complex case.} */ | |
2855 | size_t nstatus; | |
2856 | ||
2857 | /* @r{Reload the pointers.} */ | |
2858 | *inbuf = inptr; | |
2859 | outbuf = outptr; | |
2860 | ||
2861 | /* @r{Possibly reset the state.} */ | |
2862 | ||
2863 | /* @r{Redo the conversion, but this time} | |
2864 | @r{the end of the output buffer is at} | |
2865 | @r{@code{outerr}.} */ | |
2866 | @} | |
2867 | ||
2868 | /* @r{Change the status.} */ | |
2869 | status = result; | |
2870 | @} | |
2871 | else | |
2872 | /* @r{All the output is consumed, we can make} | |
2873 | @r{ another run if everything was ok.} */ | |
976780fd UD |
2874 | if (status == __GCONV_FULL_OUTPUT) |
2875 | status = __GCONV_OK; | |
390955cb UD |
2876 | @} |
2877 | @} | |
976780fd | 2878 | while (status == __GCONV_OK); |
390955cb UD |
2879 | |
2880 | /* @r{We finished one use of this step.} */ | |
976780fd | 2881 | ++data->__invocation_counter; |
390955cb UD |
2882 | @} |
2883 | ||
2884 | return status; | |
2885 | @} | |
2886 | @end smallexample | |
2887 | @end deftypevr | |
2888 | ||
2889 | This information should be sufficient to write new modules. Anybody | |
2890 | doing so should also take a look at the available source code in the GNU | |
2891 | C library sources. It contains many examples of working and optimized | |
2892 | modules. |