From: Tomek Mrugalski Date: Mon, 13 Feb 2017 19:01:58 +0000 (+0100) Subject: [5112] Changes after review: X-Git-Tag: trac4070_base~1^2 X-Git-Url: http://git.ipfire.org/gitweb.cgi?a=commitdiff_plain;h=d350d5fc2c0bf394cdb4dcc8d2d03b822c419dcf;p=thirdparty%2Fkea.git [5112] Changes after review: - lines wrapped to 100 columns (according to our coding guidelines) - explained what @3 means in bison - created link to wiki page about regenerating parser code - added terminology page that explains what CA and D2 are - corrected order of the files in Makefile.am --- diff --git a/doc/Makefile.am b/doc/Makefile.am index e009d2ac55..c00f981175 100644 --- a/doc/Makefile.am +++ b/doc/Makefile.am @@ -1,11 +1,12 @@ SUBDIRS = guide design EXTRA_DIST = version.ent.in Doxyfile Doxyfile-xml +EXTRA_DIST += devel/bison.dox EXTRA_DIST += devel/config-backend.dox EXTRA_DIST += devel/contribute.dox EXTRA_DIST += devel/mainpage.dox +EXTRA_DIST += devel/terminology.dox EXTRA_DIST += devel/unit-tests.dox -EXTRA_DIST += devel/bison.dox nobase_dist_doc_DATA = examples/ddns/sample1.json nobase_dist_doc_DATA += examples/ddns/template.json diff --git a/doc/devel/bison.dox b/doc/devel/bison.dox index a7c38b2e84..d978e0a7b7 100644 --- a/doc/devel/bison.dox +++ b/doc/devel/bison.dox @@ -9,50 +9,44 @@ @section parserIntro Parser background -Kea's data format of choice is JSON (defined in https://tools.ietf.org/html/rfc7159), which -is used in configuration files, in the command channel and also when -communicating between the DHCP servers and the DHCP-DDNS component. It is almost certain -to be used as the data format for any new features. +Kea's data format of choice is JSON (defined in https://tools.ietf.org/html/rfc7159), which is used +in configuration files, in the command channel and also when communicating between the DHCP servers +and the DHCP-DDNS component. It is almost certain to be used as the data format for any new +features. Historically, Kea used the @ref isc::data::Element::fromJSON and @ref -isc::data::Element::fromJSONFile methods to parse data expected -to be in JSON syntax. This in-house parser was developed back in the early days of -Kea when it was part of BIND 10. -Its main advantages were that it didn't have any external dependencies -and that it was already available in the source tree when Kea development -started. On the other hand, it was very difficult to modify (several attempts to -implement more robust comments had failed) and lacked a number of features. Also, it -was a pure JSON parser, so accepted anything as long as the content was correct -JSON. (This caused some problems: for example, the syntactic checks were conducted late in the -parsing process, by which time some of the information, e.g. line numbers, was no longer -available. To print meaningful error messages, the Kea team had to develop a -way to store filename, line and column information. Unfortunately this gave rise to other problems -such as data duplication.) The output from these parsers was a tree of @ref -isc::data::Element objects using shared pointers. This part of the processing we -can refer to as phase 1. - -The Element tree was then processed by set of dedicated parsers. Each parser -was able to handle its own context, e.g. global, subnet list, subnet, pool -etc. This step took the tree generated in phase 1, parsed it and -generateda an output configuration (e.g. @ref isc::dhcp::SrvConfig) or dynamic -structures (e.g. isc::data::Host). During this stage, a large number of parser objects -derived from @ref isc::dhcp::DhcpConfigParser could be instantiated for each scope and -instance of data (e.g. to parse 1000 host reservation entries a thousand -dedicated parsers were created). For convenience, this step is called phase 2. - -Other issues with the old parsers are discussed here: @ref dhcpv6ConfigParserBison -(this section is focused on DHCPv6, but the same issues affected DHCPv4 and D2) -and here: http://kea.isc.org/wiki/SimpleParser. - +isc::data::Element::fromJSONFile methods to parse data expected to be in JSON syntax. This in-house +parser was developed back in the early days of Kea when it was part of BIND 10. Its main advantages +were that it didn't have any external dependencies and that it was already available in the source +tree when Kea development started. On the other hand, it was very difficult to modify (several +attempts to implement more robust comments had failed) and lacked a number of features. Also, it was +a pure JSON parser, so accepted anything as long as the content was correct JSON. (This caused some +problems: for example, the syntactic checks were conducted late in the parsing process, by which +time some of the information, e.g. line numbers, was no longer available. To print meaningful error +messages, the Kea team had to develop a way to store filename, line and column information. +Unfortunately this gave rise to other problems such as data duplication.) The output from these +parsers was a tree of @ref isc::data::Element objects using shared pointers. This part of the +processing we can refer to as phase 1. + +The Element tree was then processed by set of dedicated parsers. Each parser was able to handle its +own context, e.g. global, subnet list, subnet, pool etc. This step took the tree generated in phase +1, parsed it and generated an output configuration (e.g. @ref isc::dhcp::SrvConfig) or dynamic +structures (e.g. isc::data::Host). During this stage, a large number of parser objects derived from +@ref isc::dhcp::DhcpConfigParser could be instantiated for each scope and instance of data (e.g. to +parse 1000 host reservation entries a thousand dedicated parsers were created). For convenience, +this step is called phase 2. + +Other issues with the old parsers are discussed here: @ref dhcpv6ConfigParserBison (this section is +focused on DHCPv6, but the same issues affected DHCPv4 and D2) and here: +http://kea.isc.org/wiki/SimpleParser. @section parserBisonIntro Flex/Bison Based Parser -To solve the issue of phase 1 mentioned earlier, a new parser has been developed -that is based on the "flex and "bison" tools. The following text uses DHCPv6 as an -example, but the same principle applies to DHCPv4 and D2; CA will likely to -follow. The new parser consists of two core elements with a wrapper around them. -The following descriptions are slightly oversimplified in order to convey the intent; -a more detailed description is available in subsequent sections. +To solve the issue of phase 1 mentioned earlier, a new parser has been developed that is based on +the "flex and "bison" tools. The following text uses DHCPv6 as an example, but the same principle +applies to DHCPv4 and D2; CA will likely to follow. The new parser consists of two core elements +with a wrapper around them. The following descriptions are slightly oversimplified in order to +convey the intent; a more detailed description is available in subsequent sections. -# Flex lexical analyzer (src/bin/dhcp6/dhcp6_lexer.ll): this is essentially a set of regular expressions and C++ code that creates new tokens that represent whatever @@ -68,13 +62,12 @@ a more detailed description is available in subsequent sections. create a token SOCKET_TYPE and pass to it its current location (that's the file name, line and column numbers). --# Bison grammar (src/bin/dhcp6/dhcp6_parser.yy): the module that defines the syntax. - Grammar and syntax are perhaps fancy words, but they simply define what is - allowed and where. Bison grammar starts with a list of tokens. Those tokens - are defined only by name ("here's the list of possible tokens that could - appear"). What constitutes a token is actually defined in the lexer. The - grammar define how the incoming tokens are expected to fall into their - places together. Let's take an example of the following input text: +-# Bison grammar (src/bin/dhcp6/dhcp6_parser.yy): the module that defines the syntax. Grammar and + syntax are perhaps fancy words, but they simply define what is allowed and where. Bison grammar + starts with a list of tokens. Those tokens are defined only by name ("here's the list of possible + tokens that could appear"). What constitutes a token is actually defined in the lexer. The + grammar define how the incoming tokens are expected to fall into their places together. Let's + take an example of the following input text: @code { "Dhcp6": @@ -83,33 +76,28 @@ a more detailed description is available in subsequent sections. } } @endcode - The lexer would generate the following sequence of tokens: LCURLY_BRACKET, - DHCP6, COLON, LCURLY_BRACKET, RENEW_TIMER, COLON, INTEGER - (a token with a value of 100), RCURLY_BRACKET, RCURLY_BRACKET, END. The - bison grammar recognises that the sequence forms a valid sentence and that - there are no errors and act upon it. (Whereas if the left and - right braces in the above example were exchanged, the bison - module would identify the sequence as syntactically incorrect.) - --# Parser context. As there is some information that needs to be passed between - parser and lexer, @ref isc::dhcp::Parser6Context is a convenience wrapper - around those two bundled together. It also works as a nice encapsulation, - hiding all the flex/bison details underneath. + The lexer would generate the following sequence of tokens: LCURLY_BRACKET, DHCP6, COLON, + LCURLY_BRACKET, RENEW_TIMER, COLON, INTEGER (a token with a value of 100), RCURLY_BRACKET, + RCURLY_BRACKET, END. The bison grammar recognises that the sequence forms a valid sentence and + that there are no errors and act upon it. (Whereas if the left and right braces in the above + example were exchanged, the bison module would identify the sequence as syntactically incorrect.) + +-# Parser context. As there is some information that needs to be passed between parser and lexer, + @ref isc::dhcp::Parser6Context is a convenience wrapper around those two bundled together. It + also works as a nice encapsulation, hiding all the flex/bison details underneath. @section parserBuild Building Flex/Bison Code -The only input file used by flex is the .ll file and the only input file used by -bison is the .yy file. When making changes to the lexer or parser, only those -two files are edited. When processed, the two tools generate a number of -.h, .hh and .cc files. The major ones have the same name as their .ll and .yy -counterparts (e.g. dhcp6_lexer.cc, dhcp6_parser.cc and dhcp6_parser.h etc.), but -a number of additional files are also created: location.hh, position.hh and -stack.hh. Those are internal bison headers that are needed for compilation. +The only input file used by flex is the .ll file and the only input file used by bison is the .yy +file. When making changes to the lexer or parser, only those two files are edited. When processed, +the two tools generate a number of .h, .hh and .cc files. The major ones have the same name as their +.ll and .yy counterparts (e.g. dhcp6_lexer.cc, dhcp6_parser.cc and dhcp6_parser.h etc.), but a +number of additional files are also created: location.hh, position.hh and stack.hh. Those are +internal bison headers that are needed for compilation. -To avoid the need for every user to have flex and bison installed, the output files -are generated when the .ll or .yy files are altered and are stored in the -Kea repository. To generate those files, issue the following sequence of -commands from the top-level Kea directory: +To avoid the need for every user to have flex and bison installed, the output files are generated +when the .ll or .yy files are altered and are stored in the Kea repository. To generate those files, +issue the following sequence of commands from the top-level Kea directory: @code ./configure --enable-generate-parser @@ -117,77 +105,69 @@ cd src/bin/dhcp6 make parser @endcode -Strictly speaking, the comment "make parser" is not necessary. If you updated -the .ll or .yy file, the -regular "make" command should pick those changes up. However, since one source -file generates multiple output files and you are likely to be using a multi-process -build (by specifying the "-j" switch on the "make" command), there may be odd side effects: -explicitly rebuilding the files manually by using "make parser" avoids any trouble. - -One problem brought on by use of flex/bison is tool version dependency. If one developer -uses version A of those tools and another developer uses B, the files -generated by the different version may be significantly different. This causes -all sorts of problems, e.g. coverity/cpp-check issues may appear and disappear: -in short, it can cause all sorts of general unhappiness. -To avoid those problems, the Kea team generates the flex/bison -files on a dedicated machine. +Strictly speaking, the comment "make parser" is not necessary. If you updated the .ll or .yy file, +the regular "make" command should pick those changes up. However, since one source file generates +multiple output files and you are likely to be using a multi-process build (by specifying the "-j" +switch on the "make" command), there may be odd side effects: explicitly rebuilding the files +manually by using "make parser" avoids any trouble. + +One problem brought on by use of flex/bison is tool version dependency. If one developer uses +version A of those tools and another developer uses B, the files generated by the different version +may be significantly different. This causes all sorts of problems, e.g. coverity/cpp-check issues +may appear and disappear: in short, it can cause all sorts of general unhappiness. To avoid those +problems, the Kea team generates the flex/bison files on a dedicated machine. See KeaRegen page +on ISC internal wiki for details. @section parserFlex Flex Detailed -Earlier sections described the lexer in a bit of an over-simplified way. The .ll file -contains a number of elements in addition to the regular expressions -and they're not as simple as was described. - -The file starts with a number of sections separated by percent (%) signs. Depending -on which section code is written in, it may be interpreted by flex, copied -verbatim to the output .cc file, copied to the output .h file or copied to both. - -There is an initial section that defines flex options. These are somewhat -documented, but the documentation for it may be a bit cryptic. When developing new -parsers, it's best to start by copying whatever we have for DHCPv6 and tweak as -needed. - -Next comes the flex conditions. They are defined with %%x and they define a -state of the lexer. A good example of a state may be comment. Once the lexer -detects that a comment's beginning, it switches to a certain condition (by calling -BEGIN(COMMENT) for example) and the code then ignores whatever follows -(especially strings that look like valid tokens) until the comment is closed -(when it returns to the default condition by calling BEGIN(INITIAL)). This is -something that is not frequently used and the only use cases for it are the -forementioned comments and file inclusions. - -After this come the syntactic contexts. Let's assume we have a parser that uses an -"ip-address" regular expression (regexp) that would return the IP_ADDRESS token. Whenever we want to -allow "ip-address", the grammar allows the IP_ADDRESS token to appear. When the -lexer is called, it will match the regexp, generate the IP_ADDRESS token and -the parser will carry out its duty. This works fine as long as you have very -specific grammar that defines everything. Sadly, that's not the case in DHCP as -we have hooks. Hook libraries can have parameters that are defined by third -party developers and they can pick whatever parameter names they want, including -"ip-address". Another example could be Dhcp4 and Dhcp6 configurations defined in a -single file. The grammar defining "Dhcp6" main contain a clause that says -"Dhcp4" may contain any generic JSON. However, the lexer may find the -"ip-address" string in the "Dhcp4" configuration and will say that it's not a part of generic JSON, but a -dedicated IP_ADDRESS token instead. The parser will then complain and the whole thing -would end up in failure. It was to solve this problem that syntactic contexts were introduced. -They tell the lexer whether input strings have specific or generic meaning. -For example, when parsing host reservations, -the lexer is expected to report the IP_ADDRESS token if "ip-address" is detected. However, when parsing generic -JSON, upon encountering "ip-address" it should return a STRING with a value of "ip-address". The list of all -contexts is enumerated in @ref isc::dhcp::Parser6Context::ParserContext. +Earlier sections described the lexer in a bit of an over-simplified way. The .ll file contains a +number of elements in addition to the regular expressions and they're not as simple as was +described. + +The file starts with a number of sections separated by percent (%) signs. Depending on which section +code is written in, it may be interpreted by flex, copied verbatim to the output .cc file, copied to +the output .h file or copied to both. + +There is an initial section that defines flex options. These are somewhat documented, but the +documentation for it may be a bit cryptic. When developing new parsers, it's best to start by +copying whatever we have for DHCPv6 and tweak as needed. + +Next comes the flex conditions. They are defined with %%x and they define a state of the lexer. A +good example of a state may be comment. Once the lexer detects that a comment's beginning, it +switches to a certain condition (by calling BEGIN(COMMENT) for example) and the code then ignores +whatever follows (especially strings that look like valid tokens) until the comment is closed (when +it returns to the default condition by calling BEGIN(INITIAL)). This is something that is not +frequently used and the only use cases for it are the forementioned comments and file inclusions. + +After this come the syntactic contexts. Let's assume we have a parser that uses an "ip-address" +regular expression (regexp) that would return the IP_ADDRESS token. Whenever we want to allow +"ip-address", the grammar allows the IP_ADDRESS token to appear. When the lexer is called, it will +match the regexp, generate the IP_ADDRESS token and the parser will carry out its duty. This works +fine as long as you have very specific grammar that defines everything. Sadly, that's not the case +in DHCP as we have hooks. Hook libraries can have parameters that are defined by third party +developers and they can pick whatever parameter names they want, including "ip-address". Another +example could be Dhcp4 and Dhcp6 configurations defined in a single file. The grammar defining +"Dhcp6" main contain a clause that says "Dhcp4" may contain any generic JSON. However, the lexer may +find the "ip-address" string in the "Dhcp4" configuration and will say that it's not a part of +generic JSON, but a dedicated IP_ADDRESS token instead. The parser will then complain and the whole +thing would end up in failure. It was to solve this problem that syntactic contexts were introduced. +They tell the lexer whether input strings have specific or generic meaning. For example, when +parsing host reservations, the lexer is expected to report the IP_ADDRESS token if "ip-address" is +detected. However, when parsing generic JSON, upon encountering "ip-address" it should return a +STRING with a value of "ip-address". The list of all contexts is enumerated in @ref +isc::dhcp::Parser6Context::ParserContext. For a DHCPv6-specific description of the conflict avoidance, see @ref dhcp6ParserConflicts. @section parserGrammar Bison Grammar -Bison has much better documentation than flex. Its latest version seems to be -available here: https://www.gnu.org/software/bison/manual. Bison is a LALR(1) -parser, which essentially means that it is able to parse (separate and analyze) -any text that is described by set of rules. You can see the more formal -description here: https://en.wikipedia.org/wiki/LALR_parser, but the plain -English explanation is that you define a set of rules and bison will walk -through input text trying to match the content to those rules. While doing -so, it will be allowed to peek at most one symbol (token) ahead. +Bison has much better documentation than flex. Its latest version seems to be available here: +https://www.gnu.org/software/bison/manual. Bison is a LALR(1) parser, which essentially means that +it is able to parse (separate and analyze) any text that is described by set of rules. You can see +the more formal description here: https://en.wikipedia.org/wiki/LALR_parser, but the plain English +explanation is that you define a set of rules and bison will walk through input text trying to match +the content to those rules. While doing so, it will be allowed to peek at most one symbol (token) +ahead. As an example, let's take a closer look at the bison grammar we have for DHCPv6. It is defined in src/bin/dhcp6/dhcp6_parser.yy. Here's a simplified excerpt: @@ -232,27 +212,23 @@ renew_timer: RENEW_TIMER COLON INTEGER; // Many other definitions follow. @endcode -The code above defines parameters that may appear in the Dhcp6 object -declaration. One important trick to understand is understand the way to handle -variable number of parameters. In bison it is most convenient to present them as -recursive lists: in this example, global_params defined in a way that allows any number of -global_param instances allowing the grammar to be easily extensible. If one -needs to add a new global parameter, just add it to the -global_param list. +The code above defines parameters that may appear in the Dhcp6 object declaration. One important +trick to understand is understand the way to handle variable number of parameters. In bison it is +most convenient to present them as recursive lists: in this example, global_params defined in a way +that allows any number of global_param instances allowing the grammar to be easily extensible. If +one needs to add a new global parameter, just add it to the global_param list. -This type of definition has several levels, each representing logical -structure of the configuration data. We start with global scope, then step -into a Dhcp6 object that has a Subnet6 list, which in turn has Subnet6 instances, -each of which has pools list and so on. Each level is represented as a separate -rule. +This type of definition has several levels, each representing logical structure of the configuration +data. We start with global scope, then step into a Dhcp6 object that has a Subnet6 list, which in +turn has Subnet6 instances, each of which has pools list and so on. Each level is represented as a +separate rule. -The "leaf" rules (that don't contain any other rules) must be defined by a -series of tokens. An example of such a rule is renew_timer, above. It is defined -as a series of 3 tokens: RENEW_TIMER, COLON and INTEGER. +The "leaf" rules (that don't contain any other rules) must be defined by a series of tokens. An +example of such a rule is renew_timer, above. It is defined as a series of 3 tokens: RENEW_TIMER, +COLON and INTEGER. -Speaking of integers, it is worth noting that some tokens can have values. Those -values are defined using %token clause. For example, dhcp6_parser.yy contains the -following: +Speaking of integers, it is worth noting that some tokens can have values. Those values are defined +using %token clause. For example, dhcp6_parser.yy contains the following: @code %token STRING "constant string" @@ -261,17 +237,15 @@ following: %token BOOLEAN "boolean" @endcode -The first line says that the token STRING has a type of std::string and when -referring to this token in error messages, it should be printed as "constant -string". +The first line says that the token STRING has a type of std::string and when referring to this token +in error messages, it should be printed as "constant string". -In principle, it is valid to define just the grammar without any corresponding -C++ code to it. Bison will go through the whole input text, match the -rules and will either say the input adhered to the rules (parsing successful) -or not (parsing failed). This may be a useful step when developing new parser, -but it has no practical value. To perform specific actions, bison allows -the injection of C++ code at almost any poing. For example we could augment the -parsing of renew_timer with some extra code: +In principle, it is valid to define just the grammar without any corresponding C++ code to it. Bison +will go through the whole input text, match the rules and will either say the input adhered to the +rules (parsing successful) or not (parsing failed). This may be a useful step when developing new +parser, but it has no practical value. To perform specific actions, bison allows the injection of +C++ code at almost any poing. For example we could augment the parsing of renew_timer with some +extra code: @code renew_timer: RENEW_TIMER { @@ -286,18 +260,18 @@ renew_timer: RENEW_TIMER { }; @endcode -This example showcases several important things. First, the ability to insert -code at almost any step is very useful. It's also a powerful debugging tool. +This example showcases several important things. First, the ability to insert code at almost any +step is very useful. It's also a powerful debugging tool. -Second, some tokens are valueless (e.g. "renew-timer" when represented as the -RENEW_TIMER token has no value), but some have values. In particular, the INTEGER -token has value which can be extracted by $ followed by a number that -represents its order, so $3 means "a value of third token or action -in this rule". +Second, some tokens are valueless (e.g. "renew-timer" when represented as the RENEW_TIMER token has +no value), but some have values. In particular, the INTEGER token has value which can be extracted +by $ followed by a number that represents its order, so $3 means "a value of third token or action +in this rule". If needed, the location of specific token (filename, line and column) can be +accessed with @ followed by a number that represents token number, e.g. @3 in the example above +returns exact location of INTEGER token. -Also, some rules may have values. This is not used often, but there are specific -cases when it's convenient. Let's take a look at the following excerpt from -dhcp6_parser.yy: +Also, some rules may have values. This is not used often, but there are specific cases when it's +convenient. Let's take a look at the following excerpt from dhcp6_parser.yy: @code ncr_protocol: NCR_PROTOCOL { @@ -313,47 +287,42 @@ ncr_protocol_value: ; @endcode -(The numbers in brackets at the end of some lines do not appear in the code; -they are used identify the statements in the following discussion.) +(The numbers in brackets at the end of some lines do not appear in the code; they are used identify +the statements in the following discussion.) -The "ncr-protocol" parameter accepts one of two values: either tcp or -udp. To handle such a case, we first enter the NCR_PROTOCOL context to tell the -lexer that we're in this scope. The lexer will then know that any incoming string of -text that is either "UDP" or "TCP" should be represented as one of the TCP or UDP -tokens. The parser knows that after NCR_PROTOCOL there will be a colon followed -by an ncr_protocol_value. The rule for ncr_protocol_value says it can be either the -TCP token or the UDP token. Let's assume the input text is: +The "ncr-protocol" parameter accepts one of two values: either tcp or udp. To handle such a case, we +first enter the NCR_PROTOCOL context to tell the lexer that we're in this scope. The lexer will then +know that any incoming string of text that is either "UDP" or "TCP" should be represented as one of +the TCP or UDP tokens. The parser knows that after NCR_PROTOCOL there will be a colon followed by an +ncr_protocol_value. The rule for ncr_protocol_value says it can be either the TCP token or the UDP +token. Let's assume the input text is: @code "ncr-protocol": "TCP" @endcode -Here's how the parser will handle it. First, it will attempt to match the rule -for ncr_protocol. It will discover the first token is NCR_PROTOCOL. As a result, -it will run the code (1), which will tell lexer to parse incoming tokens -as ncr protocol values. The next token is expected to be COLON and the one -after that the ncr_protocol_value. The lexer has already been switched into the NCR_PROTOCOL -context, so it will recognize "TCP" as TCP token, not as a string with a value of "TCP". -The parser will receive that token and match the line (2), which creates an appropriate -representation that will be used as the rule's value ($$). Finally, the parser -will unroll back to ncr_protocol rule and execute the code in lines (3) and (4). -Line (3) picks the value set up in line (2) and adds it to the stack of -values. Finally, line (4) tells the lexer that we finished the NCR protocol -parsing and it can go back to whatever state it was before. +Here's how the parser will handle it. First, it will attempt to match the rule for ncr_protocol. It +will discover the first token is NCR_PROTOCOL. As a result, it will run the code (1), which will +tell lexer to parse incoming tokens as ncr protocol values. The next token is expected to be COLON +and the one after that the ncr_protocol_value. The lexer has already been switched into the +NCR_PROTOCOL context, so it will recognize "TCP" as TCP token, not as a string with a value of +"TCP". The parser will receive that token and match the line (2), which creates an appropriate +representation that will be used as the rule's value ($$). Finally, the parser will unroll back to +ncr_protocol rule and execute the code in lines (3) and (4). Line (3) picks the value set up in +line (2) and adds it to the stack of values. Finally, line (4) tells the lexer that we finished the +NCR protocol parsing and it can go back to whatever state it was before. @section parserBisonStack Generating the Element Tree in Bison -The bison parser keeps matching rules until it reaches the end of input file. During -that process, the code needs to build a hierarchy (a tree) of inter-connected -Element objects that represents the parsed text. @ref isc::data::Element has a -complex structure that defines parent-child relation differently depending on -the type of parent (ae.g. a map and a list refer to their children in different ways). This -requires the code to be aware of the parent content. In general, every time a -new scope (an opening curly bracket in input text) is encountered, the code -pushes new Element to the stack (see @ref isc::dhcp::Parser6Context::stack_) -and every time the scope closes (a closing curly bracket in input text) the -element is removed from the stack. With this approach, we always have access -to the parent element as it's the last element on the stack. For example, when -parsing preferred-lifetime, the code does the following: +The bison parser keeps matching rules until it reaches the end of input file. During that process, +the code needs to build a hierarchy (a tree) of inter-connected Element objects that represents the +parsed text. @ref isc::data::Element has a complex structure that defines parent-child relation +differently depending on the type of parent (ae.g. a map and a list refer to their children in +different ways). This requires the code to be aware of the parent content. In general, every time a +new scope (an opening curly bracket in input text) is encountered, the code pushes new Element to +the stack (see @ref isc::dhcp::Parser6Context::stack_) and every time the scope closes (a closing +curly bracket in input text) the element is removed from the stack. With this approach, we always +have access to the parent element as it's the last element on the stack. For example, when parsing +preferred-lifetime, the code does the following: @code preferred_lifetime: PREFERRED_LIFETIME COLON INTEGER { @@ -362,50 +331,44 @@ preferred_lifetime: PREFERRED_LIFETIME COLON INTEGER { } @endcode -The first line creates an instance of IntElement with a value of the token. The -second line adds it to the current map (current = the last on the stack). This -approach has a very nice property of being generic. This rule can be referenced -from both global and subnet scope (and possibly other scopes as well) and the code -will add the IntElement object to whatever is last on the stack, be it global, -subnet or perhaps even something else (maybe one day we will allow preferred -lifetime to be defined on a per pool or per host basis?). +The first line creates an instance of IntElement with a value of the token. The second line adds it +to the current map (current = the last on the stack). This approach has a very nice property of +being generic. This rule can be referenced from both global and subnet scope (and possibly other +scopes as well) and the code will add the IntElement object to whatever is last on the stack, be it +global, subnet or perhaps even something else (maybe one day we will allow preferred lifetime to be +defined on a per pool or per host basis?). @section parserSubgrammar Parsing a Partial Configuration -All the explanations so far assumed that we're operating in a default case of -receiving the configuration as a whole. That is the case during startup and -reconfiguration. However, both DHCPv4 and DHCPv6 support certain cases when the -input text is not the whole configuration, but rather certain parts of it. There -are several examples of such cases. The most common are unit-tests. They -typically don't have the outermost { } or Dhcp6 object, but simply define -whatever parameters are being tested. Second, we have the command channel that will, -in the near future, contain parts of the configuration, depending on the -command. For example, "add-reservation" will contain a host reservation only. - -Bison by default does not support multiple start rules, but there's a trick -that can provide such a capability. The trick assumes that the starting -rule may allow one of the artificial tokens that represent the scope -expected. For example, when called from the "add-reservation" command, the -artificial token will be SUB_RESERVATION and it will trigger the parser -to bypass the global braces { and } and the "Dhcp6" token and jump immediately to the sub_reservation. - -This trick is also implemented in the lexer. A flag called start_token_flag, -when initially set to true, will cause the lexer to emit an artificial -token once, before parsing any input whatsoever. - -This optional feature can be skipped altogether if you don't plan to parse parts -of the configuration. +All the explanations so far assumed that we're operating in a default case of receiving the +configuration as a whole. That is the case during startup and reconfiguration. However, both DHCPv4 +and DHCPv6 support certain cases when the input text is not the whole configuration, but rather +certain parts of it. There are several examples of such cases. The most common are unit-tests. They +typically don't have the outermost { } or Dhcp6 object, but simply define whatever parameters are +being tested. Second, we have the command channel that will, in the near future, contain parts of +the configuration, depending on the command. For example, "add-reservation" will contain a host +reservation only. + +Bison by default does not support multiple start rules, but there's a trick that can provide such a +capability. The trick assumes that the starting rule may allow one of the artificial tokens that +represent the scope expected. For example, when called from the "add-reservation" command, the +artificial token will be SUB_RESERVATION and it will trigger the parser to bypass the global braces +{ and } and the "Dhcp6" token and jump immediately to the sub_reservation. + +This trick is also implemented in the lexer. A flag called start_token_flag, when initially set to +true, will cause the lexer to emit an artificial token once, before parsing any input whatsoever. + +This optional feature can be skipped altogether if you don't plan to parse parts of the +configuration. @section parserBisonExtend Extending the Grammar -Adding new parameters to existing parsers is very easy once you get hold of the -concept of what the grammar rules represent. The first step is to understand -where the parameter is to be allowed. Typically a new parameter is allowed -in one scope and only over time is it added to other scopes. Recently support -for a 4o6-interface-id parameter has been added. That is a parameter that can -be defined in a subnet and takes a string argument. You can see the actual -change conducted in this commit: -(https://github.com/isc-projects/kea/commit/9fccdbf54c4611dc10111ad8ff96d36cad59e1d6). +Adding new parameters to existing parsers is very easy once you get hold of the concept of what the +grammar rules represent. The first step is to understand where the parameter is to be +allowed. Typically a new parameter is allowed in one scope and only over time is it added to other +scopes. Recently support for a 4o6-interface-id parameter has been added. That is a parameter that +can be defined in a subnet and takes a string argument. You can see the actual change conducted in +this commit: (https://github.com/isc-projects/kea/commit/9fccdbf54c4611dc10111ad8ff96d36cad59e1d6). Here's the complete set of changes that were necessary. @@ -427,9 +390,9 @@ Here's the complete set of changes that were necessary. } } @endcode - It tells the parser that when in Subnet4 context, an incoming "4o6-interface-id" string - should be represented as the SUBNET_4O6_INTERFACE_ID token. In any other context, - it should be represented as a string. + It tells the parser that when in Subnet4 context, an incoming "4o6-interface-id" string should be + represented as the SUBNET_4O6_INTERFACE_ID token. In any other context, it should be represented + as a string. 3. Add the rule that will define the value. A user is expected to add something like @code @@ -445,11 +408,10 @@ Here's the complete set of changes that were necessary. ctx.leave(); }; @endcode - Here's a good example of the context use. We have no idea what sort of interface-id - the user will use. Typically that will be an integer, but it may be something - weird that happens to match our reserved keywords. Therefore we switch to - no keyword context. This tells the lexer to interpret everything as string, - integer or float. + Here's a good example of the context use. We have no idea what sort of interface-id the user will + use. Typically that will be an integer, but it may be something weird that happens to match our + reserved keywords. Therefore we switch to no keyword context. This tells the lexer to interpret + everything as string, integer or float. 4. Finally, extend the existing subnet4_param that defines all allowed parameters in the Subnet4 scope to also cover our new parameter (the new line marked with *): diff --git a/doc/devel/mainpage.dox b/doc/devel/mainpage.dox index 77e50c156b..20f3b83430 100644 --- a/doc/devel/mainpage.dox +++ b/doc/devel/mainpage.dox @@ -119,6 +119,7 @@ * - @subpage cplSignals * * @section miscellaneousTopics Miscellaneous Topics + * - @subpage terminology * - @subpage parser * - @subpage logKeaLogging * - @subpage logBasicIdeas diff --git a/doc/devel/terminology.dox b/doc/devel/terminology.dox new file mode 100644 index 0000000000..11750ff20e --- /dev/null +++ b/doc/devel/terminology.dox @@ -0,0 +1,28 @@ +// Copyright (C) 2017 Internet Systems Consortium, Inc. ("ISC") +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this +// file, You can obtain one at http://mozilla.org/MPL/2.0/. + +/** +@page terminology Terminology + +This page explains some common abbreviations and terms: + +- CA - Control Agent. That's a separate module that talks with Kea DHCPv4, DHCPv6 (and soon also D2) + over control channel and exposes their internal commands using RESTful interface. + +- D2 - This is a nickname of DHCP-Dynamic DNS server module. Since using the full name is awkward, + we often use shortened version of it: D2. + +- DHCP - Dynamic Host Configuration Protocol. There are two similar, but operationally different + protocols: DHCPv4 and DHCPv6. When v4 or v6 is not specified, the DHCP expression applies to both. + +- DHCPv4 - Dynamic Host Configuration Protocol for IPv4, a protocol that defines how IPv4 hosts can + obtain IPv4 addresses and other configuration from the servers. Defined in RFC2131. + +- DHCPv6 - Dynamic Host Configuration Protocol for IPv6, a protocol that defines how IPv6 hosts and + router can obtain IPv6 addresses, IPv6 prefixes and other configuration from the servers. Defined + in RFC3315. + +*/