From: Alan T. DeKok Date: Wed, 3 Nov 2021 16:22:30 +0000 (-0400) Subject: Revert "Revert "remove the 'instantiate' section, as it's no longer necessary"" X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=00d8d85bca84274671947a3afe8e885fa9b2842a;p=thirdparty%2Ffreeradius-server.git Revert "Revert "remove the 'instantiate' section, as it's no longer necessary"" This reverts commit 09416b43dfcb6b232bd184fd3a56156a5dd4426a. --- diff --git a/doc/antora/modules/installation/pages/upgrade.adoc b/doc/antora/modules/installation/pages/upgrade.adoc index 4a5928ffd1b..69a01c9cee6 100644 --- a/doc/antora/modules/installation/pages/upgrade.adoc +++ b/doc/antora/modules/installation/pages/upgrade.adoc @@ -64,6 +64,23 @@ subsection there. See also xref:raddb/templates.conf.adoc[`templates.conf`] for a way to regain one global configuration for `Access-Request` packets. +=== Instantiate Section + +The `instantiate` section has been removed. It originally started out +as a way to ensure that modules were instantiated in a particular +order. As of 3.0.9, listing modules in the `instantiate` section was +no longer necessary. The functionality was left in version 3 for +compatibility, but has now been removed from version 4. + +The other use of the `instantiate` section was to define "virtual" +modules for dynamic expansion. That functionality has been moved to +the `mods-available/` and `mods-enabled/` directories. i.e. in +version 4, just list the virtual module in a file, as if it was a real +module. + +See the xref:mods-available/redundantsql.adoc[redundant_sql module] +for more information. + == Virtual Servers There are some changes to the virtual servers in `v4`. First, every diff --git a/doc/antora/modules/raddb/nav.adoc b/doc/antora/modules/raddb/nav.adoc index b522eddf810..59da9df1ab7 100644 --- a/doc/antora/modules/raddb/nav.adoc +++ b/doc/antora/modules/raddb/nav.adoc @@ -57,6 +57,7 @@ *** xref:mods-available/redis.adoc[REDIS Module] *** xref:mods-available/redis_ippool.adoc[Redis IP Pool Module] *** xref:mods-available/rediswho.adoc[REDISWho Module] +*** xref:mods-available/redundant_sql.adoc[redundant_sql Module] *** xref:mods-available/rest.adoc[Rest Module] *** xref:mods-available/mruby.adoc[Ruby Module] *** xref:mods-available/smbpasswd.adoc[SMBPasswd Module] diff --git a/doc/antora/modules/raddb/pages/mods-available/redundant_sql.adoc b/doc/antora/modules/raddb/pages/mods-available/redundant_sql.adoc new file mode 100644 index 00000000000..dd3fa2d23ff --- /dev/null +++ b/doc/antora/modules/raddb/pages/mods-available/redundant_sql.adoc @@ -0,0 +1,55 @@ + + + + += redundant_sql Module + +The `redundant_sql` module handles SQL expansions in a redundant manner. + + +This configuration can be thought of as a `virtual` module. + +e.g. If you have two redundant SQL servers, and you want to use +them in the authorize and accounting sections, you could place a +`redundant` block in each section, containing the exact same text. +Or, you could uncomment the following lines, and list +`redundant_sql` in the authorize and accounting sections. + +The `virtual` module defined here can also be used with dynamic +expansions, under a few conditions: + + * The section is one of `group`, `redundant`, `load-balance`, or + `redundant-load-balance` + * The section contains module names ONLY, and no sub-sections + * All modules in the section are using the same + driver, e.g. They are all sql, or all ldap, etc. + +When those conditions are satisfied, the server will +automatically register a dynamic expansion, using the +name of the `virtual` module. In the example below, +it will be `redundant_sql`. You can then use this expansion +just like any other: + + update reply { + Filter-Id := "%{redundant_sql: ... }" + } + +In this example, the expansion is done via module `sql1`, and if +that expansion fails, using module `sql2`. + +For best results, configure the `pool` subsection of the module so +that `retry_delay` is non-zero. That will allow the redundant +block to quickly ignore all "down" SQL databases. If instead we +have `retry_delay = 0`, then every time the redundant block is +used, the server will try to open a connection to every `down` +database, causing problems. + + +== Default Configuration + +``` +redundant redundant_sql { + sql1 + sql2 +} +``` diff --git a/doc/antora/modules/raddb/pages/radiusd.conf.adoc b/doc/antora/modules/raddb/pages/radiusd.conf.adoc index 054b2b5f323..66552024706 100644 --- a/doc/antora/modules/raddb/pages/radiusd.conf.adoc +++ b/doc/antora/modules/raddb/pages/radiusd.conf.adoc @@ -477,10 +477,33 @@ num_networks:: Only one network thread is supported for now. num_workers:: The worker threads can be varied. It should be -at least one, and no more than 32. Since each request is +at least one, and no more than 128. Since each request is non-blocking, there is no reason to run hundreds of threads as in v3. +When set to 0 the number of workers will be set from the number +of cores available on the system. + + + +openssl_async_pool_init:: Controls the initial number of async +contexts that are allocated when a worker thread is created. +One async context is required for every TLS session (every +RADSEC connection, every TLS based method still in progress). + + + +openssl_async_pool_max:: Controls the maximum number of async +contexts which are allocated to a worker thread. +If the maximum is reached, then no more TLS sessions can be +created. + +Note: Setting this to 0 will mean unlimited async contexts +will be created. But as of 3.0.0, OpenSSL has no mechanism +to shrink the async pool. This means if there's a +significant traffic spike the process will continue to use +large amounts of memory until it's restarted. + .SNMP notifications. @@ -714,8 +737,10 @@ security { } $INCLUDE clients.conf thread pool { - num_networks = 1 - num_workers = 4 +# num_networks = 1 + num_workers = 0 +# openssl_async_pool_init = 64 +# openssl_async_pool_max = 1024 } #$INCLUDE trigger.conf modules { diff --git a/raddb/mods-available/redundant_sql b/raddb/mods-available/redundant_sql new file mode 100644 index 00000000000..bb1c90135f0 --- /dev/null +++ b/raddb/mods-available/redundant_sql @@ -0,0 +1,53 @@ +# -*- text -*- +# +# +# $Id$ + +####################################################################### +# +# = redundant_sql Module +# +# The `redundant_sql` module handles SQL expansions in a redundant manner. +# +# +# This configuration can be thought of as a `virtual` module. +# +# e.g. If you have two redundant SQL servers, and you want to use +# them in the authorize and accounting sections, you could place a +# `redundant` block in each section, containing the exact same text. +# Or, you could uncomment the following lines, and list +# `redundant_sql` in the authorize and accounting sections. +# +# The `virtual` module defined here can also be used with dynamic +# expansions, under a few conditions: +# +# * The section is one of `group`, `redundant`, `load-balance`, or +# `redundant-load-balance` +# * The section contains module names ONLY, and no sub-sections +# * All modules in the section are using the same +# driver, e.g. They are all sql, or all ldap, etc. +# +# When those conditions are satisfied, the server will +# automatically register a dynamic expansion, using the +# name of the `virtual` module. In the example below, +# it will be `redundant_sql`. You can then use this expansion +# just like any other: +# +# update reply { +# Filter-Id := "%{redundant_sql: ... }" +# } +# +# In this example, the expansion is done via module `sql1`, and if +# that expansion fails, using module `sql2`. +# +# For best results, configure the `pool` subsection of the module so +# that `retry_delay` is non-zero. That will allow the redundant +# block to quickly ignore all "down" SQL databases. If instead we +# have `retry_delay = 0`, then every time the redundant block is +# used, the server will try to open a connection to every `down` +# database, causing problems. +# +redundant redundant_sql { + sql1 + sql2 +} diff --git a/raddb/radiusd.conf.in b/raddb/radiusd.conf.in index 450638cd76a..9cdc67f05a2 100644 --- a/raddb/radiusd.conf.in +++ b/raddb/radiusd.conf.in @@ -624,82 +624,6 @@ modules { $INCLUDE mods-enabled/ } -# -# .Instantiation -# -# This section orders the loading of the modules. Modules -# listed here will get loaded BEFORE the later sections like -# authorize, authenticate, etc. get examined. -# -# This section is not strictly needed. When a section like -# authorize refers to a module, it's automatically loaded and -# initialized. However, some modules may not be listed in any -# of the following sections, so they can be listed here. -# -# Also, listing modules here ensures that you have control over -# the order in which they are initialized. If one module needs -# something defined by another module, you can list them in order -# here, and ensure that the configuration will be OK. -# -# After the modules listed here have been loaded, all of the modules -# in the "mods-enabled" directory will be loaded. Loading the -# "mods-enabled" directory means that unlike Version 2, you usually -# don't need to list modules here. -# -instantiate { - # - # We list the counter module here so that it registers - # the check_name attribute before any module which sets - # it. - # -# dailycounter - - # - # subsections here can be thought of as `virtual` modules. - # - # e.g. If you have two redundant SQL servers, and you want to - # use them in the authorize and accounting sections, you could - # place a `redundant` block in each section, containing the - # exact same text. Or, you could uncomment the following - # lines, and list `redundant_sql` in the authorize and - # accounting sections. - # - # The `virtual` module defined here can also be used with - # dynamic expansions, under a few conditions: - # - # * The section is `redundant`, or `load-balance`, or - # `redundant-load-balance` - # * The section contains modules ONLY, and no sub-sections - # * All modules in the section are using the same rlm_ - # driver, e.g. They are all sql, or all ldap, etc. - # - # When those conditions are satisfied, the server will - # automatically register a dynamic expansion, using the - # name of the `virtual` module. In the example below, - # it will be `redundant_sql`. You can then use this expansion - # just like any other: - # - # update reply { - # Filter-Id := "%{redundant_sql: ... }" - # } - # - # In this example, the expansion is done via module `sql1`, - # and if that expansion fails, using module `sql2`. - # - # For best results, configure the `pool` subsection of the - # module so that `retry_delay` is non-zero. That will allow - # the redundant block to quickly ignore all "down" SQL - # databases. If instead we have `retry_delay = 0`, then - # every time the redundant block is used, the server will try - # to open a connection to every `down` database, causing - # problems. - # -# redundant redundant_sql { -# sql1 -# sql2 -# } -} - ###################################################################### # # .Policies diff --git a/src/lib/server/base.c b/src/lib/server/base.c index 63ed54ad847..35ba2ef2b45 100644 --- a/src/lib/server/base.c +++ b/src/lib/server/base.c @@ -82,7 +82,7 @@ int server_init(CONF_SECTION *cs) /* * Instantiate the modules */ - if (modules_instantiate() < 0) return -1; + if (modules_instantiate(cs) < 0) return -1; /* * Call xlat instantiation functions (after the xlats have been compiled) diff --git a/src/lib/server/module.c b/src/lib/server/module.c index 59000c1e7ba..4c7666be794 100644 --- a/src/lib/server/module.c +++ b/src/lib/server/module.c @@ -59,6 +59,8 @@ static fr_cmd_table_t cmd_module_table[]; static int _module_instantiate(void *instance); +static int virtual_module_instantiate(CONF_SECTION *vm_cs); + /* * Ordered by component */ @@ -1327,13 +1329,16 @@ static int _module_instantiate(void *instance) * Allows the module to initialise connection pools, and complete any registrations that depend on * attributes created during the bootstrap phase. * + * @param[in] root of the server configuration. * @return * - 0 on success. * - -1 on failure. */ -int modules_instantiate(void) +int modules_instantiate(CONF_SECTION *root) { - void *instance; + void *instance; + CONF_ITEM *ci; + CONF_SECTION *modules; fr_rb_iter_inorder_t iter; DEBUG2("#### Instantiating modules ####"); @@ -1346,6 +1351,32 @@ int modules_instantiate(void) } } + modules = cf_section_find(root, "modules", NULL); + if (!modules) return 0; + + /* + * Instantiate the virtual modules. + */ + for (ci = cf_item_next(modules, NULL); + ci != NULL; + ci = cf_item_next(modules, ci)) { + char const *name; + CONF_SECTION *subcs; + + if (!cf_item_is_section(ci)) continue; + + subcs = cf_item_to_section(ci); + + /* + * If it's not an unlang keyword, then skip it. + * It must be a module we already checked. + */ + name = cf_section_name1(subcs); + if (!unlang_compile_is_keyword(name)) continue; + + if (virtual_module_instantiate(subcs) < 0) return -1; + } + return 0; } @@ -1574,23 +1605,23 @@ module_instance_t *module_bootstrap(module_instance_t const *parent, CONF_SECTIO return mi; } -/** Bootstrap a virtual module from an instantiate section +/** Instantiate a virtual module from an instantiate section * - * @param[in] vm_cs that defines the virtual module. + * @param[in] cs that defines the virtual module. * @return * - 0 on success. * - -1 on failure. */ -static int virtual_module_bootstrap(CONF_SECTION *vm_cs) +static int virtual_module_instantiate(CONF_SECTION *cs) { char const *name; bool all_same = true; module_t const *last = NULL; CONF_ITEM *sub_ci = NULL; CONF_PAIR *cp; - module_instance_t *instance; + module_instance_t *mi; - name = cf_section_name1(vm_cs); + name = cf_section_name1(cs); /* * Groups, etc. must have a name. @@ -1599,29 +1630,43 @@ static int virtual_module_bootstrap(CONF_SECTION *vm_cs) (strcmp(name, "redundant") == 0) || (strcmp(name, "redundant-load-balance") == 0) || (strcmp(name, "load-balance") == 0)) { - name = cf_section_name2(vm_cs); + name = cf_section_name2(cs); if (!name) { - cf_log_err(vm_cs, "Subsection must have a name"); + cf_log_err(cs, "Keyword module must have a second name"); return -1; } - if (unlang_compile_is_keyword(name)) { - is_reserved: - cf_log_err(vm_cs, "Virtual modules cannot overload unlang keywords"); - return -1; - } + /* + * name2 was already checked in modules_bootstrap() + */ + fr_assert(!unlang_compile_is_keyword(name)); } else { - goto is_reserved; + cf_log_err(cs, "Module names cannot be unlang keywords '%s'", name); + return -1; + } + + /* + * Ensure that the module doesn't exist. + */ + mi = module_by_name(NULL, name); + if (mi) { + ERROR("Duplicate module \"%s\" in file %s[%d] and file %s[%d]", + name, + cf_filename(cs), + cf_lineno(cs), + cf_filename(mi->dl_inst->conf), + cf_lineno(mi->dl_inst->conf)); + return -1; } /* * Ensure that the modules we reference here exist. */ - while ((sub_ci = cf_item_next(vm_cs, sub_ci))) { + while ((sub_ci = cf_item_next(cs, sub_ci))) { if (cf_item_is_pair(sub_ci)) { cp = cf_item_to_pair(sub_ci); if (cf_pair_value(cp)) { - cf_log_err(sub_ci, "Cannot set return codes in a %s block", cf_section_name1(vm_cs)); + cf_log_err(sub_ci, "Cannot set return codes in a %s block", cf_section_name1(cs)); return -1; } @@ -1630,17 +1675,17 @@ static int virtual_module_bootstrap(CONF_SECTION *vm_cs) * * Note that we don't care what the method is, just that it exists. */ - instance = module_by_name_and_method(NULL, NULL, NULL, NULL, cf_pair_attr(cp)); - if (!instance) { + mi = module_by_name_and_method(NULL, NULL, NULL, NULL, cf_pair_attr(cp)); + if (!mi) { cf_log_err(sub_ci, "Module instance \"%s\" referenced in %s block, does not exist", - cf_pair_attr(cp), cf_section_name1(vm_cs)); + cf_pair_attr(cp), cf_section_name1(cs)); return -1; } if (all_same) { if (!last) { - last = instance->module; - } else if (last != instance->module) { + last = mi->module; + } else if (last != mi->module) { last = NULL; all_same = false; } @@ -1657,7 +1702,7 @@ static int virtual_module_bootstrap(CONF_SECTION *vm_cs) /* * Register a redundant xlat */ - if (all_same && (xlat_register_legacy_redundant(vm_cs) < 0)) return -1; + if (all_same && (xlat_register_legacy_redundant(cs) < 0)) return -1; return 0; } @@ -1674,7 +1719,7 @@ static int virtual_module_bootstrap(CONF_SECTION *vm_cs) */ int modules_bootstrap(CONF_SECTION *root) { - CONF_ITEM *ci, *next; + CONF_ITEM *ci; CONF_SECTION *cs, *modules; /* @@ -1698,17 +1743,18 @@ int modules_bootstrap(CONF_SECTION *root) */ for (ci = cf_item_next(modules, NULL); ci != NULL; - ci = next) { + ci = cf_item_next(modules, ci)) { char const *name; CONF_SECTION *subcs; module_instance_t *instance; - next = cf_item_next(modules, ci); - if (!cf_item_is_section(ci)) continue; subcs = cf_item_to_section(ci); + /* + * name2 can't be a keyword + */ name = cf_section_name2(subcs); if (name && unlang_compile_is_keyword(name)) { invalid_name: @@ -1717,7 +1763,17 @@ int modules_bootstrap(CONF_SECTION *root) } name = cf_section_name1(subcs); - if (unlang_compile_is_keyword(name)) goto invalid_name; + + /* + * For now, ignore name1 which is a keyword. + */ + if (unlang_compile_is_keyword(name)) { + if (!cf_section_name2(subcs)) { + cf_log_err(subcs, "Missing second name at '%s'", name); + return -1; + } + continue; + } /* * Skip inline templates, and disallow "template { ... }" @@ -1729,56 +1785,6 @@ int modules_bootstrap(CONF_SECTION *root) instance = module_bootstrap(NULL, subcs); if (!instance) return -1; - - if (!next || !cf_item_is_section(next)) continue; - } - - /* - * Look for the 'instantiate' section, which tells us - * the instantiation order of the modules, and also allows - * us to load modules with no authorize/authenticate/etc. - * sections. - */ - cs = cf_section_find(root, "instantiate", NULL); - if (cs) { - cf_log_debug(cs, " instantiate {"); - ci = NULL; - - /* - * Loop over the items in the 'instantiate' section. - */ - while ((ci = cf_item_next(cs, ci))) { - CONF_SECTION *vm_cs; - - /* - * Skip sections and "other" stuff. - * Sections will be handled later, if - * they're referenced at all... - */ - if (cf_item_is_pair(ci)) { - cf_log_warn(ci, "Only virtual modules can be instantiated " - "with the instantiate section"); - continue; - } - - /* - * Skip section - */ - if (!cf_item_is_section(ci)) continue; - - vm_cs = cf_item_to_section(ci); - cf_log_debug(ci, "Instantiating virtual module \"%s %s\"", - cf_section_name1(vm_cs), cf_section_name2(vm_cs)); - - /* - * Can only be "redundant" or - * "load-balance" or - * "redundant-load-balance" - */ - if (virtual_module_bootstrap(cf_item_to_section(ci)) < 0) return -1; - } - - cf_log_debug(cs, " }"); } cf_log_debug(modules, " } # modules"); diff --git a/src/lib/server/module.h b/src/lib/server/module.h index 83e47131a1f..dae4b63f983 100644 --- a/src/lib/server/module.h +++ b/src/lib/server/module.h @@ -334,7 +334,7 @@ int modules_thread_instantiate(TALLOC_CTX *ctx, fr_event_list_t *el) CC_HINT(no void modules_thread_detach(void); -int modules_instantiate(void) CC_HINT(nonnull); +int modules_instantiate(CONF_SECTION *root) CC_HINT(nonnull); module_instance_t *module_bootstrap(module_instance_t const *parent, CONF_SECTION *cs) CC_HINT(nonnull(2));