[thirdparty/gcc.git] / gcc / doc / cppinternals / overview.rst

..
  Copyright 1988-2022 Free Software Foundation, Inc.
  This is part of the GCC manual.
  For copying conditions, see the copyright.rst file.

Overview
********

The lexer is contained in the file :samp:`lex.cc`.  It is a hand-coded
lexer, and not implemented as a state machine.  It can understand C, C++
and Objective-C source code, and has been extended to allow reasonably
successful preprocessing of assembly language.  The lexer does not make
an initial pass to strip out trigraphs and escaped newlines, but handles
them as they are encountered in a single pass of the input file.  It
returns preprocessing tokens individually, not a line at a time.

It is mostly transparent to users of the library, since the library's
interface for obtaining the next token, ``cpp_get_token``, takes care
of lexing new tokens, handling directives, and expanding macros as
necessary.  However, the lexer does expose some functionality so that
clients of the library can easily spell a given token, such as
``cpp_spell_token`` and ``cpp_token_len``.  These functions are
useful when generating diagnostics, and for emitting the preprocessed
output.
Commit	Line	Data
c63539ff ML	1	..
	2	Copyright 1988-2022 Free Software Foundation, Inc.
	3	This is part of the GCC manual.
	4	For copying conditions, see the copyright.rst file.
	5
	6	Overview
	7	********
	8
	9	The lexer is contained in the file :samp:`lex.cc`. It is a hand-coded
	10	lexer, and not implemented as a state machine. It can understand C, C++
	11	and Objective-C source code, and has been extended to allow reasonably
	12	successful preprocessing of assembly language. The lexer does not make
	13	an initial pass to strip out trigraphs and escaped newlines, but handles
	14	them as they are encountered in a single pass of the input file. It
	15	returns preprocessing tokens individually, not a line at a time.
	16
	17	It is mostly transparent to users of the library, since the library's
	18	interface for obtaining the next token, ``cpp_get_token``, takes care
	19	of lexing new tokens, handling directives, and expanding macros as
	20	necessary. However, the lexer does expose some functionality so that
	21	clients of the library can easily spell a given token, such as
	22	``cpp_spell_token`` and ``cpp_token_len``. These functions are
	23	useful when generating diagnostics, and for emitting the preprocessed
3ed1b4ce	24	output.