Added the full documentation set for the OpenTelemetry filter.
The main README served as the user-facing guide covering build
instructions, core OpenTelemetry concepts, the complete filter
configuration reference, usage examples with worked scenarios,
CLI commands, and known limitations.
Supplementary documents provided a detailed configuration guide with
worked examples (README-configuration), an internal C structure reference
for developers (README-conf), a function reference organized by source
file (README-func), an architecture and implementation review
(README-implementation), and miscellaneous notes (README-misc).
--- /dev/null
+ -----------------------------------------
+ The HAProxy OpenTelemetry filter (OTel)
+ Version 1.0
+ ( Last update: 2026-03-18 )
+ -----------------------------------------
+ Author : Miroslav Zagorac
+ Contact : mzagorac at haproxy dot com
+
+
+SUMMARY
+--------
+
+ 0. Terms
+ 1. Introduction
+ 2. Build instructions
+ 3. Basic concepts in OpenTelemetry
+ 4. OTel configuration
+ 4.1. OTel scope
+ 4.2. "otel-instrumentation" section
+ 4.3. "otel-scope" section
+ 4.4. "otel-group" section
+ 5. Examples
+ 6. OTel CLI
+ 7. Known bugs and limitations
+
+
+0. Terms
+---------
+
+* OTel: The HAProxy OpenTelemetry filter
+
+OTel is the HAProxy filter that allows you to send telemetry data (traces,
+metrics and logs) to observability backends via the OpenTelemetry protocol.
+
+
+1. Introduction
+----------------
+
+Nowadays there is a growing need to divide a process into microservices and
+there is a problem of monitoring the work of the same process. One way to solve
+this problem is to use a distributed tracing service in a central location.
+
+The OTel filter is the successor to the OpenTracing (OT) filter and is built on
+the OpenTelemetry standard, which unifies distributed tracing, metrics and
+logging into a single observability framework. Unlike the older OpenTracing
+filter which relied on vendor-specific tracer plugins, the OTel filter uses the
+OpenTelemetry protocol (OTLP) to export data directly to any compatible backend.
+
+The OTel filter is a standard HAProxy filter, so what applies to others also
+applies to this one (of course, by that I mean what is described in the
+documentation, more precisely in the doc/internals/filters.txt file).
+
+The OTel filter activation is done explicitly by specifying it in the HAProxy
+configuration. If this is not done, the OTel filter in no way participates in
+the work of HAProxy.
+
+The OTel filter allows intensive use of ACLs, which can be defined anywhere in
+the configuration. Thus, it is possible to use the filter only for those
+connections that are of interest to us.
+
+
+2. Build instructions
+----------------------
+
+OTel is the HAProxy filter and as such is compiled together with HAProxy.
+
+To communicate with an OpenTelemetry compatible backend, the OTel filter uses
+the OpenTelemetry C Wrapper library (which again uses the OpenTelemetry C++
+SDK). This means that we must have the library installed on the system on which
+we want to compile or use HAProxy.
+
+Instructions for compiling and installing the required library can be found at
+https://github.com/haproxytech/opentelemetry-c-wrapper .
+
+The OTel filter can be more easily compiled using the pkg-config tool, if we
+have the OpenTelemetry C Wrapper library installed so that it contains
+pkg-config files (which have the .pc extension). If the pkg-config tool cannot
+be used, then the path to the directory where the include files and libraries
+are located can be explicitly specified.
+
+Below are examples of the two ways to compile HAProxy with the OTel filter, the
+first using the pkg-config tool and the second explicitly specifying the path to
+the OpenTelemetry C Wrapper include and library.
+
+Note: prompt '%' indicates that the command is executed under an unprivileged
+ user, while prompt '#' indicates that the command is executed under the
+ root user.
+
+Example of compiling HAProxy using the pkg-config tool (assuming the
+OpenTelemetry C Wrapper library is installed in the /opt directory):
+
+ % PKG_CONFIG_PATH=/opt/lib/pkgconfig make -j8 USE_OTEL=1 TARGET=linux-glibc
+
+The OTel filter can also be compiled in debug mode as follows:
+
+ % PKG_CONFIG_PATH=/opt/lib/pkgconfig make -j8 USE_OTEL=1 OTEL_DEBUG=1 TARGET=linux-glibc
+
+HAProxy compilation example explicitly specifying path to the OpenTelemetry C
+Wrapper include and library:
+
+ % make -j8 USE_OTEL=1 OTEL_INC=/opt/include OTEL_LIB=/opt/lib TARGET=linux-glibc
+
+In case we want to use debug mode, then it looks like this:
+
+ % make -j8 USE_OTEL=1 OTEL_DEBUG=1 OTEL_INC=/opt/include OTEL_LIB=/opt/lib TARGET=linux-glibc
+
+To enable OpenTelemetry context propagation via HAProxy variables (in addition
+to HTTP headers), add the OTEL_USE_VARS=1 option:
+
+ % PKG_CONFIG_PATH=/opt/lib/pkgconfig make -j8 USE_OTEL=1 OTEL_USE_VARS=1 TARGET=linux-glibc
+
+If the library we want to use is not installed on a unix system, then a locally
+installed library can be used (say, which is compiled and installed in the user
+home directory). In this case instead of /opt/include and /opt/lib the
+equivalent paths to the local installation should be specified. Of course, in
+that case the pkg-config tool can also be used if we have a complete
+installation (with .pc files).
+
+Last but not least, if the pkg-config tool is not used when compiling, then the
+HAProxy executable may not be able to find the OpenTelemetry C Wrapper library
+at startup. This can be solved in several ways, for example using the
+LD_LIBRARY_PATH environment variable which should be set to the path where the
+library is located before starting the HAProxy.
+
+ % LD_LIBRARY_PATH=/opt/lib /path-to/haproxy ...
+
+Another way is to add RUNPATH to HAProxy executable that contains the path to
+the library in question.
+
+ % make -j8 USE_OTEL=1 OTEL_RUNPATH=1 OTEL_INC=/opt/include OTEL_LIB=/opt/lib TARGET=linux-glibc
+
+After HAProxy is compiled, we can check if the OTel filter is enabled:
+
+ % ./haproxy -vv | grep opentelemetry
+ --- command output ----------
+ [ OTel] opentelemetry
+ --- command output ----------
+
+A summary of all OTel build options:
+
+ USE_OTEL - enable the OpenTelemetry filter
+ OTEL_DEBUG - compile the filter in debug mode
+ OTEL_INC - force path to opentelemetry-c-wrapper include files
+ OTEL_LIB - force path to opentelemetry-c-wrapper library
+ OTEL_RUNPATH - add opentelemetry-c-wrapper RUNPATH to executable
+ OTEL_USE_VARS - enable context propagation via HAProxy variables
+
+
+3. Basic concepts in OpenTelemetry
+-----------------------------------
+
+Basic concepts of OpenTelemetry can be read on the OpenTelemetry documentation
+website https://opentelemetry.io/docs/concepts/ .
+
+Here we will list only the most important elements of distributed tracing.
+
+A 'trace' is a description of the complete transaction we want to record in the
+tracing system. A 'span' is an operation that represents a unit of work that is
+recorded in a tracing system. A 'span context' is a group of information
+related to a particular span that is passed on to the system (from service to
+service). Using this context, we can add new spans to already open trace (or
+supplement data in already open spans).
+
+An individual span may contain one or more attributes, events, links and baggage
+items.
+
+An 'attribute' is a key-value element that is valid for the entire span.
+Attributes describe properties of the span such as HTTP method, URL, status
+code, and so on.
+
+A span 'event' is a named key-value element that allows you to write some data
+at a certain time within the span's lifetime. It can be used for debugging or
+recording notable occurrences.
+
+A 'link' is a reference to another span (possibly in a different trace) that is
+causally related to the current span. Unlike the parent-child relationship,
+links represent non-hierarchical associations between spans.
+
+A 'baggage' item is a key-value data pair that can be used for the duration of
+an entire trace, from the moment it is added to the span.
+
+A span 'status' indicates the outcome of the operation: unset (default), ok
+(successful) or error (failed). An optional description string can accompany
+the error status.
+
+
+4. OTel configuration
+----------------------
+
+In order for the OTel filter to be used, the 'insecure-fork-wanted' keyword
+must be set in the HAProxy 'global' section. This is required because the
+OpenTelemetry C++ SDK creates background threads for data export and batch
+processing. HAProxy will refuse to load the configuration if this keyword
+is missing.
+
+ global
+ insecure-fork-wanted
+ ...
+
+The OTel filter must also be included in the HAProxy configuration, in the
+proxy section (frontend / listen / backend):
+
+ frontend otel-test
+ ...
+ filter opentelemetry [id <id>] config <file>
+ ...
+
+If no filter id is specified, 'otel-filter' is used as default. The 'config'
+parameter must be specified and it contains the path of the OTel filter
+configuration file. This file defines the OTel scopes, groups and
+instrumentation sections (see section 4.1). The YAML configuration for the
+OpenTelemetry SDK is a separate file, referenced by the 'config' keyword inside
+the "otel-instrumentation" section (see section 4.2).
+
+
+4.1 OTel scope
+---------------
+
+If the filter id is defined for the OTel filter, then the OTel scope with the
+same name should be defined in the configuration file. In the same
+configuration file we can have several defined OTel scopes.
+
+Each OTel scope must have a defined (only one) "otel-instrumentation" section
+that is used to configure the operation of the OTel filter and define the used
+groups and scopes.
+
+OTel scope starts with the id of the filter specified in square brackets and
+ends with the end of the file or when a new OTel scope is defined.
+
+For example, this defines two OTel scopes in the same configuration file:
+ [my-first-otel-filter]
+ otel-instrumentation instrumentation1
+ ...
+ otel-group group1
+ ...
+ otel-scope scope1
+ ...
+
+ [my-second-otel-filter]
+ ...
+
+
+4.2. "otel-instrumentation" section
+-------------------------------------
+
+Only one "otel-instrumentation" section must be defined for each OTel scope.
+
+The mandatory 'config' keyword defines the YAML configuration file for the
+OpenTelemetry SDK. This file specifies the telemetry pipeline: exporters,
+processors, samplers, providers and signals.
+
+Through optional keywords can be defined ACLs, logging, rate limit, and groups
+and scopes that define the tracing model.
+
+
+otel-instrumentation <name>
+ A new OTel instrumentation with the name <name> is created.
+
+ Arguments :
+ name - the name of the OpenTelemetry instrumentation section
+
+
+ The following keywords are supported in this section:
+ - mandatory keywords:
+ - config
+
+ - optional keywords:
+ - acl
+ - debug-level
+ - groups
+ - [no] log
+ - [no] option disabled
+ - [no] option dontlog-normal
+ - [no] option hard-errors
+ - rate-limit
+ - scopes
+
+
+acl <aclname> <criterion> [flags] [operator] <value> ...
+ Declare or complete an access list.
+
+ To configure and use the ACL, see section 7 of the HAProxy Configuration
+ Manual.
+
+
+config <file>
+ The mandatory keyword associated with the OTel instrumentation configuration.
+ This keyword sets the path of the YAML configuration file for the
+ OpenTelemetry SDK. The YAML file defines the complete telemetry pipeline
+ including exporters, samplers, processors, providers and signal routing.
+
+ The YAML configuration file supports the following top-level sections:
+
+ 'exporters' - defines telemetry data destinations. Supported exporter types
+ are:
+ - otlp_grpc : export via OTLP over gRPC
+ - otlp_http : export via OTLP over HTTP (JSON or Protobuf)
+ - otlp_file : export to local files in OTLP format
+ - zipkin : export to Zipkin-compatible backends
+ - elasticsearch : export to Elasticsearch
+ - ostream : write to a file (text output, useful for debugging)
+ - memory : in-memory buffer (useful for testing)
+
+ 'samplers' - defines trace sampling strategies. Supported types:
+ - always_on : sample every trace
+ - always_off : sample no traces
+ - trace_id_ratio_based : sample a fraction of traces (set by ratio)
+ - parent_based : sampling decision based on parent span
+
+ 'processors' - defines how telemetry data is processed before export:
+ - batch : batch spans before exporting (configurable queue size, export
+ interval and batch size)
+ - single : export each span individually
+
+ 'readers' - defines metric readers with configurable export interval and
+ timeout.
+
+ 'providers' - defines resource attributes (service name, version, instance ID,
+ namespace, etc.) that are attached to all telemetry data.
+
+ 'signals' - binds the above components together for each signal type (traces,
+ metrics, logs), specifying which exporter, sampler, processor, reader and
+ provider to use.
+
+ Arguments :
+ file - the path of the YAML configuration file
+
+
+debug-level <value>
+ This keyword sets the value of the debug level related to the display of debug
+ messages in the OTel filter. The 'debug-level' value is a bitmask, ie a
+ single value bit enables or disables the display of the corresponding debug
+ message that uses that bit. The default value is set via the
+ FLT_OTEL_DEBUG_LEVEL macro in the include/config.h file. Debug level value is
+ used only if the OTel filter is compiled with the debug mode enabled,
+ otherwise it is ignored.
+
+ Arguments :
+ value - bitmask value (hexadecimal notation, e.g. 0x77f)
+
+
+groups <name> ...
+ A list of "otel-group" groups used for the currently defined instrumentation
+ is declared. Several groups can be specified in one line.
+
+ Arguments :
+ name - the name of the OTel group
+
+
+log global
+log <addr> [len <len>] [format <fmt>] <facility> [<level> [<minlevel>]]
+no log
+ Enable per-instance logging of events and traffic.
+
+ To configure and use the logging system, see section 4.2 of the HAProxy
+ Configuration Manual.
+
+
+option disabled
+no option disabled
+ Keyword which turns the operation of the OTel filter on or off. By default
+ the filter is on.
+
+
+option dontlog-normal
+no option dontlog-normal
+ Enable or disable logging of normal, successful processing. By default, this
+ option is disabled. For this option to be considered, logging must be turned
+ on.
+
+ See also: 'log' keyword description.
+
+
+option hard-errors
+no option hard-errors
+ During the operation of the filter, some errors may occur, caused by incorrect
+ configuration of the instrumentation or some error related to the operation of
+ HAProxy. By default, such an error will not interrupt the filter operation
+ for the stream in which the error occurred. If the 'hard-errors' option is
+ enabled, the operation error prohibits all further processing of events and
+ groups in the stream in which the error occurred.
+
+
+rate-limit <value>
+ This option allows limiting the use of the OTel filter, ie it can be
+ influenced whether the OTel filter is activated for a stream or not.
+ Determining whether or not a filter is activated depends on the value of this
+ option that is compared to a randomly selected value when attaching the filter
+ to the stream. By default, the value of this option is set to 100.0, ie the
+ OTel filter is activated for each stream.
+
+ Arguments :
+ value - floating point value ranging from 0.0 to 100.0
+
+
+scopes <name> ...
+ This keyword declares a list of "otel-scope" definitions used for the
+ currently defined instrumentation. Multiple scopes can be specified in the
+ same line.
+
+ Arguments :
+ name - the name of the OTel scope
+
+
+4.3. "otel-scope" section
+--------------------------
+
+Stream processing begins with filter attachment, then continues with the
+processing of a number of defined events and groups, and ends with filter
+detachment. The "otel-scope" section is used to define actions related to
+individual events. However, this section may be part of a group, so the event
+does not have to be part of the definition.
+
+
+otel-scope <name>
+ Creates a new OTel scope definition named <name>.
+
+ Arguments :
+ name - the name of the OTel scope
+
+
+ The following keywords are supported in this section:
+ - acl
+ - attribute
+ - baggage
+ - event
+ - extract
+ - finish
+ - idle-timeout
+ - inject
+ - instrument
+ - link
+ - log-record
+ - otel-event
+ - span
+ - status
+
+
+acl <aclname> <criterion> [flags] [operator] <value> ...
+ Declare or complete an access list.
+
+ To configure and use the ACL, see section 7 of the HAProxy Configuration
+ Manual.
+
+
+attribute <key> <sample> ...
+ This keyword allows setting an attribute for the currently active span. The
+ first argument is the name of the attribute (key) and the rest are its value.
+ A value can consist of one or more sample expressions. If the value is only
+ one sample, then the type of that data depends on the type of the HAProxy
+ sample. If the value contains more samples, then the data type is string.
+ The data conversion table is below:
+
+ HAProxy sample data type | the OpenTelemetry data type
+ --------------------------+----------------------------
+ NULL | NULL
+ BOOL | BOOL
+ INT32 | INT64
+ UINT32 | UINT64
+ INT64 | INT64
+ UINT64 | UINT64
+ IPV4 | STRING
+ IPV6 | STRING
+ STRING | STRING
+ BINARY | UNSUPPORTED
+ --------------------------+----------------------------
+
+ Arguments :
+ key - key part of a data pair (attribute name)
+ sample - sample expression (value part of a data pair), at least
+ one sample must be present
+
+
+baggage <key> <sample> ...
+ Baggage items allow the propagation of data between spans, ie allow the
+ assignment of metadata that is propagated to future children spans. This data
+ is formatted in the style of key-value pairs and is part of the context that
+ can be transferred between processes that are part of a server architecture.
+
+ This keyword allows setting the baggage for the currently active span. The
+ data type is always a string, ie any sample type is converted to a string.
+ The exception is a binary value that is not supported by the OTel filter.
+
+ See the 'attribute' keyword description for the data type conversion table.
+
+ Arguments :
+ key - key part of a data pair
+ sample - sample expression (value part of a data pair), at least one sample
+ must be present
+
+
+event <name> <key> <sample> ...
+ This keyword allows adding a span event to the currently active span. A span
+ event is a named, timestamped annotation with optional attributes. The data
+ type is always a string, ie any sample type is converted to a string.
+
+ See the 'attribute' keyword description for the data type conversion table.
+
+ Arguments :
+ name - name of the span event
+ key - key part of a data pair (attribute name within the event)
+ sample - sample expression (value part of a data pair), at least one sample
+ must be present
+
+
+extract <name-prefix> [use-vars | use-headers]
+ For a more detailed description of the propagation process of the span
+ context, see the description of the keyword 'inject'. Only the process of
+ extracting data from the carrier is described here.
+
+ The default carrier is HTTP headers. If OTEL_USE_VARS is enabled at compile
+ time, the 'use-vars' option can be used instead to extract context from
+ HAProxy variables.
+
+ Arguments :
+ name-prefix - data name prefix (ie key element prefix)
+ use-vars - data is extracted from HAProxy variables
+ use-headers - data is extracted from the HTTP header
+
+
+ Below is an example of using HAProxy variables to transfer span context data:
+
+ --- test/ctx/otel.cfg -----------------------------------------------
+ ...
+ otel-scope client_session_start_2
+ extract "otel_ctx_1" use-vars
+ span "Client session" parent "otel_ctx_1"
+ ...
+ ---------------------------------------------------------------------
+
+
+finish <name> ...
+ Closing a particular span or span context. Instead of the name of the span,
+ there are several specially predefined names with which we can finish certain
+ groups of spans. So it can be used as the name '*req*' for all open spans
+ related to the request channel, '*res*' for all open spans related to the
+ response channel and '*' for all open spans regardless of which channel they
+ are related to. Several spans and/or span contexts can be specified in one
+ line.
+
+ Arguments :
+ name - the name of the span or span context
+
+
+inject <name-prefix> [use-vars] [use-headers]
+ In OpenTelemetry, the transfer of data related to the tracing process between
+ microservices that are part of a larger service is done through the
+ propagation of the span context. The basic operations that allow us to access
+ and transfer this data are 'inject' and 'extract'.
+
+ 'inject' allows us to extract span context so that the obtained data can be
+ forwarded to another process (microservice) via the selected carrier. 'inject'
+ in the name actually means inject data into carrier. Carrier is an interface
+ here (ie a data structure) that allows us to transfer tracing state from one
+ process to another.
+
+ Data transfer can take place via one of two selected storage methods, the
+ first is by adding data to the HTTP header and the second is by using HAProxy
+ variables (the latter requires OTEL_USE_VARS=1 at compile time). Only data
+ transfer via HTTP header can be used to transfer data to another process (ie
+ microservice). All data is organized in the form of key-value data pairs.
+
+ No matter which data transfer method you use, we need to specify a prefix for
+ the key element. All alphanumerics (lowercase only) and underline character
+ can be used to construct the data name prefix. Uppercase letters can actually
+ be used, but they will be converted to lowercase when creating the prefix.
+ The special prefix '-' can be used to generate the name automatically from the
+ scope's event name or the span name.
+
+ Arguments :
+ name-prefix - data name prefix (ie key element prefix), or '-' for automatic
+ naming
+ use-vars - HAProxy variables are used to store and transfer data
+ (requires OTEL_USE_VARS=1)
+ use-headers - HTTP headers are used to store and transfer data
+
+
+ Below is an example of using HTTP headers and variables to propagate the span
+ context.
+
+ --- test/ctx/otel.cfg -----------------------------------------------
+ ...
+ otel-scope client_session_start_1
+ span "HAProxy session" root
+ inject "otel_ctx_1" use-headers use-vars
+ ...
+ ---------------------------------------------------------------------
+
+ Because HAProxy does not allow the '-' character in the variable name (which
+ is automatically generated by the OpenTelemetry API and on which we have no
+ influence), it is converted to the letter 'D'. We can see that there is no
+ such conversion in the name of the HTTP header because the '-' sign is allowed
+ there. Due to this conversion, initially all uppercase letters are converted
+ to lowercase because otherwise we would not be able to distinguish whether the
+ disputed sign '-' is used or not.
+
+ Thus created HTTP headers and variables are deleted when executing the
+ 'finish' keyword or when detaching the stream from the filter.
+
+
+instrument { update <name> [<attr>] | <type> <name> [<aggr>] [<desc>] [<unit>] <value> [<bounds>] }
+ This keyword allows creating or updating metric instruments within the scope.
+ Metric instruments record numerical measurements that are exported alongside
+ traces.
+
+ To create a new instrument, specify the instrument type, a name, and a sample
+ expression providing the measurement value (preceded by the 'value' keyword).
+ Optionally, a human-readable description (preceded by 'desc') and a unit
+ string (preceded by 'unit') can be added.
+
+ An aggregation type can be specified using the 'aggr' keyword followed by one
+ of the supported aggregation types listed below. When specified, a metrics
+ view is registered with the given aggregation strategy. If no aggregation
+ type is specified, the SDK default is used.
+
+ For histogram instruments (hist_int), optional bucket boundaries can be
+ specified using the 'bounds' keyword followed by a double-quoted string of
+ space-separated integers in strictly ascending order. When bounds are
+ specified without an explicit aggregation type, histogram aggregation is
+ used automatically.
+
+ To update an existing instrument (previously created in another scope), use
+ 'update' followed by the name of the instrument. Optional attributes can be
+ added using the 'attr' keyword followed by key-value pairs.
+
+ Supported instrument types:
+ - cnt_int : counter (uint64)
+ - hist_int : histogram (uint64)
+ - udcnt_int : up-down counter (int64)
+ - gauge_int : gauge (int64)
+
+ Supported aggregation types:
+ - drop : measurements are discarded
+ - histogram : explicit bucket histogram
+ - last_value : last recorded value
+ - sum : sum of recorded values
+ - default : SDK default for the instrument type
+ - exp_histogram : base-2 exponential histogram
+
+ Observable (asynchronous) instruments are not supported. The OpenTelemetry
+ SDK invokes their callbacks from an external background thread that is not
+ a HAProxy thread. HAProxy sample fetches rely on internal per-thread-group
+ state and return incorrect results when called from a non-HAProxy thread.
+
+ Double-precision types are not supported because HAProxy sample fetches do
+ not return double values.
+
+ For example:
+ instrument cnt_int "my_counter" desc "Counter" value int(1)
+ instrument hist_int "my_hist" aggr exp_histogram desc "Latency" value lat_ns_tot unit "ns"
+ instrument hist_int "my_hist2" desc "Latency" value lat_ns_tot unit "ns" bounds "100 1000 10000 100000"
+ instrument update "my_counter" attr "key1" "val1"
+
+ Arguments :
+ type - the instrument type (see list above)
+ name - the name of the instrument
+ aggr - optional aggregation type (see list above)
+ desc - optional human-readable description of the instrument
+ unit - optional unit string for the instrument
+ value - sample expression providing the measurement value
+ bounds - optional histogram bucket boundaries (hist_int only)
+ attr - attribute key-value pairs (update form only)
+
+
+log-record <severity> [id <integer>] [event <name>] [span <span-name>] [attr <key> <value>] ... <sample> ...
+ This keyword emits an OpenTelemetry log record within the scope. The first
+ argument is a required severity level. Optional keywords follow in any order
+ before the trailing sample expressions that form the log record body:
+
+ id <integer> - numeric event identifier
+ event <name> - event name string
+ span <span-name> - associate the log record with an open span
+ attr <key> <value> - add a key-value attribute (repeatable)
+
+ The remaining arguments at the end are sample fetch expressions. A single
+ sample preserves its native type; multiple samples are concatenated as a
+ string.
+
+ Supported severity levels follow the OpenTelemetry specification:
+ trace, trace2, trace3, trace4, debug, debug2, debug3, debug4,
+ info, info2, info3, info4, warn, warn2, warn3, warn4,
+ error, error2, error3, error4, fatal, fatal2, fatal3, fatal4
+
+ The log record is only emitted when the logger is enabled for the configured
+ severity. If a 'span' reference is given but the named span is not found at
+ runtime, the log record is emitted without span correlation.
+
+ For example:
+ log-record info str("heartbeat")
+ log-record info id 1001 event "http-request" span "Frontend HTTP request" attr "http.method" "GET" method url
+ log-record trace id 1000 event "session-start" span "Client session" attr "attr_1_key" "attr_1_value" src str(":") src_port
+ log-record warn event "server-unavailable" str("503 Service Unavailable")
+
+ Arguments :
+ severity - the log severity level (see list above)
+ id - optional numeric event identifier
+ event - optional event name
+ span - optional name of an open span to associate with
+ attr - optional attribute key-value pairs (repeatable)
+ sample - sample fetch expression(s) forming the log record body
+
+
+link <span> ...
+ This keyword adds span links to the currently active span. A span link
+ represents a causal relationship to another span without establishing a
+ parent-child hierarchy. Links are useful for connecting spans across
+ different traces or for associating related spans within the same trace.
+
+ Multiple span names can be specified in one line. Each name is resolved at
+ runtime by searching for an active span or an extracted context with that
+ name. If a referenced span or context cannot be found, the link is silently
+ skipped.
+
+ Arguments :
+ span - the name of a span or span context to link to
+
+
+otel-event <name> [{ if | unless } <condition>]
+ Set the event that triggers the 'otel-scope' to which it is assigned.
+ Optionally, it can be followed by an ACL-based condition, in which case it
+ will only be evaluated if the condition is true.
+
+ ACL-based conditions are executed in the context of a stream that processes
+ the client and server connections. To configure and use the ACL, see section
+ 7 of the HAProxy Configuration Manual.
+
+ Arguments :
+ name - the event name
+ condition - a standard ACL-based condition
+
+ Supported events are (the table gives the names of the events in the OTel
+ filter and the corresponding equivalent in the SPOE filter):
+
+ -------------------------------------|------------------------------
+ the OTel filter | the SPOE filter
+ -------------------------------------|------------------------------
+ on-stream-start | -
+ on-stream-stop | -
+ on-idle-timeout | -
+ on-backend-set | -
+ -------------------------------------|------------------------------
+ on-client-session-start | on-client-session
+ on-frontend-tcp-request | on-frontend-tcp-request
+ on-http-wait-request | -
+ on-http-body-request | -
+ on-frontend-http-request | on-frontend-http-request
+ on-switching-rules-request | -
+ on-backend-tcp-request | on-backend-tcp-request
+ on-backend-http-request | on-backend-http-request
+ on-process-server-rules-request | -
+ on-http-process-request | -
+ on-tcp-rdp-cookie-request | -
+ on-process-sticking-rules-request | -
+ on-http-headers-request | -
+ on-http-end-request | -
+ on-client-session-end | -
+ on-server-unavailable | -
+ -------------------------------------|------------------------------
+ on-server-session-start | on-server-session
+ on-tcp-response | on-tcp-response
+ on-http-wait-response | -
+ on-process-store-rules-response | -
+ on-http-response | on-http-response
+ on-http-headers-response | -
+ on-http-end-response | -
+ on-http-reply | -
+ on-server-session-end | -
+ -------------------------------------|------------------------------
+
+ --- Stream lifecycle events (not tied to a channel analyzer) ---
+
+ The on-stream-start and on-stream-stop events fire from the stream_start and
+ stream_stop filter callbacks respectively, before any channel processing
+ begins and after all channel processing ends. No channel is available at
+ that point, so context injection/extraction via HTTP headers cannot be used
+ in scopes bound to these events. Sample fetches in these scopes are not
+ direction-constrained.
+
+ The on-idle-timeout event fires periodically when the stream has no data
+ transfer activity. It requires the 'idle-timeout' keyword to set the
+ interval. This event is useful for heartbeat spans, idle-time metrics, and
+ idle-time log records. It fires from the check_timeouts filter callback
+ using HAProxy's tick-based timer infrastructure.
+
+ The on-backend-set event fires from the stream_set_backend filter callback
+ when a backend is assigned to the stream. It is not called if the frontend
+ and the backend are the same proxy.
+
+
+ --- Request channel events ---
+
+ Analyzer events (tied to AN_REQ_* bits):
+
+ The on-frontend-tcp-request event fires during frontend TCP content inspection
+ (AN_REQ_INSPECT_FE).
+
+ The on-http-wait-request event fires after the complete HTTP request has been
+ received (AN_REQ_WAIT_HTTP). This is a post-analyzer event.
+
+ The on-http-body-request event fires when the HTTP request body is available
+ for inspection (AN_REQ_HTTP_BODY).
+
+ The on-frontend-http-request event fires during frontend HTTP request
+ processing: header rules, monitoring, statistics and redirects
+ (AN_REQ_HTTP_PROCESS_FE).
+
+ The on-switching-rules-request event fires when backend switching rules are
+ evaluated (AN_REQ_SWITCHING_RULES).
+
+ The on-backend-tcp-request event fires during backend TCP content inspection
+ (AN_REQ_INSPECT_BE).
+
+ The on-backend-http-request event fires during backend HTTP request processing
+ (AN_REQ_HTTP_PROCESS_BE).
+
+ The on-process-server-rules-request event fires when use-server rules are
+ evaluated (AN_REQ_SRV_RULES).
+
+ The on-http-process-request event fires during inner HTTP request processing
+ (AN_REQ_HTTP_INNER).
+
+ The on-tcp-rdp-cookie-request event fires when RDP cookie persistence is
+ evaluated (AN_REQ_PRST_RDP_COOKIE).
+
+ The on-process-sticking-rules-request event fires when stick-table persistence
+ matching rules are evaluated (AN_REQ_STICKING_RULES).
+
+ Non-analyzer events (not tied to AN_REQ_* bits):
+
+ The on-client-session-start event fires when the request channel analysis
+ begins. It corresponds to the start of a new client session.
+
+ The on-http-headers-request event fires from the http_headers filter callback
+ after all HTTP request headers have been parsed and analyzed.
+
+ The on-http-end-request event fires from the http_end filter callback when all
+ HTTP request data has been processed and forwarded.
+
+ The on-client-session-end event fires when the request channel analysis ends.
+
+ The on-server-unavailable event fires during request channel end-analysis when
+ response analyzers were configured but never executed because the server was
+ not reached.
+
+
+ --- Response channel events ---
+
+ Analyzer events (tied to AN_RES_* bits):
+
+ The on-tcp-response event fires during TCP response content inspection
+ (AN_RES_INSPECT).
+
+ The on-http-wait-response event fires after the complete HTTP response has
+ been received (AN_RES_WAIT_HTTP). This is a post-analyzer event.
+
+ The on-process-store-rules-response event fires when stick-table store rules
+ are evaluated (AN_RES_STORE_RULES).
+
+ The on-http-response event fires during backend HTTP response processing
+ (AN_RES_HTTP_PROCESS_BE).
+
+ Non-analyzer events (not tied to AN_RES_* bits):
+
+ The on-server-session-start event fires when the response channel analysis
+ begins, after a server connection has been established.
+
+ The on-http-headers-response event fires from the http_headers filter callback
+ after all HTTP response headers have been parsed and analyzed.
+
+ The on-http-end-response event fires from the http_end filter callback when
+ all HTTP response data has been processed and forwarded.
+
+ The on-http-reply event fires from the http_reply filter callback when HAProxy
+ generates an internal reply (error page, deny response, redirect). It always
+ fires on the response channel.
+
+ The on-server-session-end event fires when the response channel analysis ends.
+
+
+idle-timeout <time>
+ Set the idle timeout interval for a scope bound to the 'on-idle-timeout'
+ event. The timer fires periodically at the given interval when the stream
+ is idle. This keyword is mandatory for scopes using the 'on-idle-timeout'
+ event and cannot be used with any other event.
+
+ The <time> argument accepts the standard HAProxy time format: a number
+ followed by a unit suffix (ms, s, m, h, d). A value of zero is not
+ permitted.
+
+ Arguments :
+ time - the idle timeout interval (e.g. 5s, 500ms, 1m)
+
+ Example :
+ scopes on_idle_timeout
+ ..
+ otel-scope on_idle_timeout
+ idle-timeout 5s
+ span "heartbeat" root
+ attribute "idle.elapsed" str("idle-check")
+ instrument cnt_int "idle.count" value int(1)
+ log-record info str("heartbeat")
+ otel-event on-idle-timeout
+
+
+span <name> [<reference>] [<link>] [root]
+ Creating a new span (or referencing an already opened one). If a new span is
+ created, it can have a parent reference to another span or context, an inline
+ link to another span, or be marked as a root span. If no reference is
+ specified, the new span will become a root span. We need to pay attention to
+ the fact that in one trace there can be only one root span. If a non-existent
+ span is specified as a reference, a new span will not be created.
+
+ The parent reference is set using the 'parent' keyword followed by the name of
+ an existing span or extracted context. An inline link is set using the 'link'
+ keyword followed by a span or context name. The 'root' keyword explicitly
+ marks the span as a root span.
+
+ For example:
+ span "HAProxy session" root
+ span "Client session" parent "HAProxy session"
+ span "HTTP request" parent "TCP request" link "HAProxy session"
+ span "Client session" parent "otel_ctx_1"
+
+ Only one inline link can be specified per 'span' declaration. For multiple
+ links, use the standalone 'link' keyword described above.
+
+ Arguments :
+ name - the name of the span being created or referenced
+ (operation name)
+ reference - 'parent <name>' or 'link <name>' or 'root'
+
+
+status <code> [<sample> ...]
+ This keyword sets the status for the currently active span. The status
+ indicates the outcome of the operation represented by the span.
+
+ The status code is one of the following predefined values:
+ - ignore : do not set any status (default)
+ - unset : explicitly mark status as unset
+ - ok : the operation completed successfully
+ - error : the operation resulted in an error
+
+ An optional description can follow the status code, consisting of one or more
+ sample expressions whose values are concatenated as a string. The description
+ is typically used with the 'error' status to provide additional context about
+ the failure.
+
+ For example:
+ status "ok"
+ status "error" str("http.status_code: ") status
+
+ Arguments :
+ code - the status code (ignore, unset, ok, error)
+ sample - optional sample expression(s) for the status description
+
+
+4.4. "otel-group" section
+--------------------------
+
+This section allows us to define a group of OTel scopes, that is not activated
+via an event but is triggered from TCP or HTTP rules. More precisely, these are
+the following rules: 'tcp-request', 'tcp-response', 'http-request',
+'http-response' and 'http-after-response'. These rules can be defined in the
+HAProxy configuration file.
+
+The action keyword used in these rules is 'otel-group', and it takes the filter
+id and the group name as arguments:
+
+ http-response otel-group <filter-id> <group-name> [{ if | unless } ...]
+
+
+otel-group <name>
+ Creates a new OTel group definition named <name>.
+
+ Arguments :
+ name - the name of the OTel group
+
+
+ The following keywords are supported in this section:
+ - scopes
+
+
+scopes <name> ...
+ 'otel-scope' sections that are part of the specified group are defined. If
+ the mentioned 'otel-scope' sections are used only in some OTel group, they do
+ not have to have defined events. Several 'otel-scope' sections can be
+ specified in one line.
+
+ Arguments :
+ name - the name of the 'otel-scope' section
+
+
+5. Examples
+------------
+
+Several examples of the OTel filter configuration can be found in the test
+directory. A brief description of the prepared configurations follows:
+
+cmp - a configuration made for comparison purposes with other tracing
+ implementations.
+
+sa - a standalone configuration in which all possible events are used. This
+ is the most comprehensive example demonstrating spans, attributes,
+ events, links, baggage, status and other features.
+
+ctx - a configuration similar to 'sa', with the difference that the spans are
+ opened using extracted span contexts as references instead of direct
+ parent span names. This demonstrates the inject/extract context
+ propagation mechanism using HAProxy variables.
+
+fe be - a more complex example of the OTel filter configuration that uses two
+ cascaded HAProxy services (frontend and backend). The span context
+ between HAProxy processes is transmitted via the HTTP header using
+ inject/extract.
+
+empty - an empty configuration in which the OTel filter is initialized but no
+ event is triggered. It is not very usable, except to check the behavior
+ of the OTel filter in the case of a similar configuration.
+
+
+The OTel filter does not use tracer plugins. Instead, telemetry data is
+exported using the OpenTelemetry protocol (OTLP) directly to any compatible
+backend. The backend is configured through the YAML configuration file
+specified by the 'config' keyword.
+
+In order to be able to collect and view trace data we need an OpenTelemetry
+compatible backend. There are many options available, including:
+
+ - Jaeger : https://www.jaegertracing.io/
+ - Grafana : https://grafana.com/oss/tempo/
+ - Zipkin : https://zipkin.io/
+ - SigNoz : https://signoz.io/
+ - Datadog : https://www.datadoghq.com/
+
+For quick testing, a simple setup using the OpenTelemetry Collector and Jaeger
+can be started with Docker:
+
+ # docker run -d --name jaeger -p 4317:4317 -p 4318:4318 -p 16686:16686 jaegertracing/all-in-one:latest
+
+This starts Jaeger with OTLP/gRPC on port 4317, OTLP/HTTP on port 4318 and the
+web UI on port 16686. If we want to use that container later, it can be started
+and stopped using the 'docker container start/stop' commands.
+
+The test configurations use a YAML file that defines an OTLP/HTTP exporter
+sending data to localhost:4318. A typical minimal YAML configuration looks like
+this:
+
+ --- otel.yml --------------------------------------------------------
+ exporters:
+ my_exporter:
+ type: otlp_http
+ endpoint: "http://localhost:4318/v1/traces"
+
+ samplers:
+ my_sampler:
+ type: always_on
+
+ processors:
+ my_processor:
+ type: batch
+
+ providers:
+ my_provider:
+ resources:
+ - service.name: "haproxy"
+
+ signals:
+ traces:
+ scope_name: "HAProxy OTel"
+ exporters: my_exporter
+ samplers: my_sampler
+ processors: my_processor
+ providers: my_provider
+ ---------------------------------------------------------------------
+
+In order to use any of the configurations from the test directory, we can run
+one of the pre-configured scripts:
+
+ % ./run-sa.sh
+ % ./run-ctx.sh
+ % ./run-cmp.sh
+ % ./run-fe-be.sh
+
+
+6. OTel CLI
+------------
+
+Via the HAProxy CLI interface we can find out the current status of the OTel
+filter and change several of its settings.
+
+All supported CLI commands can be found in the following way, using the socat
+utility with the assumption that the HAProxy CLI socket path is set to
+/tmp/haproxy.sock (of course, instead of socat, nc or other utility can be used
+with a change in arguments when running the same):
+
+ % echo "help" | socat - UNIX-CONNECT:/tmp/haproxy.sock | grep flt-otel
+ --- command output ----------
+ flt-otel debug [level] : set the OTel filter debug level
+ flt-otel disable : disable the OTel filter
+ flt-otel enable : enable the OTel filter
+ flt-otel soft-errors : turning off hard-errors mode
+ flt-otel hard-errors : enabling hard-errors mode
+ flt-otel logging [state] : set logging state
+ flt-otel rate [value] : set the rate limit
+ flt-otel status : show the OTel filter status
+ --- command output ----------
+
+'flt-otel debug' can only be used in case the OTel filter is compiled with the
+debug mode enabled. When invoked without arguments, these commands display the
+current value of the respective setting.
+
+
+7. Known bugs and limitations
+-------------------------------
+
+The name of the span context definition can contain only letters, numbers and
+characters '_' and '-'. Also, all uppercase letters in the name are converted
+to lowercase. The character '-' is converted internally to the 'D' character,
+and since a HAProxy variable is generated from that name, this should be taken
+into account if we want to use it somewhere in the HAProxy configuration. The
+above mentioned span context is used in the 'inject' and 'extract' keywords.
+
+An inline span link (using the 'link' keyword within a 'span' declaration) is
+limited to a single link per span declaration due to the fixed argument count
+(maximum 7 arguments). For multiple links, use the standalone 'link' keyword
+instead.
+
+Let's look a little at the example test/fe-be (configurations are in the
+test/fe and test/be directories, 'fe' is here the abbreviation for frontend and
+'be' for backend). In case we have the 'rate-limit' set to a value less than
+100.0, then distributed tracing will not be started with each new HTTP request.
+It also means that the span context will not be delivered (via the HTTP header)
+to the backend HAProxy process. The 'rate-limit' on the backend HAProxy must be
+set to 100.0, but because the frontend HAProxy does not send a span context
+every time, all such cases will cause an error to be reported on the backend
+server. Therefore, the 'hard-errors' option must be set on the backend server,
+so that processing on that stream is stopped as soon as the first error occurs.
--- /dev/null
+OpenTelemetry Filter Configuration Structures
+==============================================================================
+
+1 Overview
+------------------------------------------------------------------------------
+
+The OpenTelemetry filter configuration is a tree of C structures that mirrors
+the hierarchical layout of the filter's configuration file. Each structure type
+carries a common header macro, and its allocation and deallocation are performed
+by macro-generated init/free function pairs defined in conf_funcs.h.
+
+The root of the tree is flt_otel_conf, which owns the instrumentation settings,
+groups, and scopes. Scopes contain the actual tracing and metrics definitions:
+contexts, spans, instruments, and their sample expressions.
+
+Source files:
+ include/conf.h Structure definitions and debug macros.
+ include/conf_funcs.h Init/free macro templates and declarations.
+ src/conf.c Init/free implementations for all types.
+
+
+2 Common Macros
+------------------------------------------------------------------------------
+
+Two macros provide the building blocks embedded in every configuration
+structure.
+
+2.1 FLT_OTEL_CONF_STR(p)
+
+Expands to an anonymous struct containing a string pointer and its cached
+length:
+
+ struct {
+ char *p;
+ size_t p_len;
+ };
+
+Used for auxiliary string fields that do not need list linkage (e.g. ref_id and
+ctx_id in flt_otel_conf_span).
+
+2.2 FLT_OTEL_CONF_HDR(p)
+
+Expands to an anonymous struct that extends FLT_OTEL_CONF_STR with a
+configuration file line number and an intrusive list node:
+
+ struct {
+ char *p;
+ size_t p_len;
+ int cfg_line;
+ struct list list;
+ };
+
+Every configuration structure embeds FLT_OTEL_CONF_HDR as its first member.
+The <p> parameter names the identifier field (e.g. "id", "key", "str", "span",
+"fmt_expr"). The list node chains the structure into its parent's list.
+The cfg_line records the source line for error reporting.
+
+
+3 Structure Hierarchy
+------------------------------------------------------------------------------
+
+The complete ownership tree, from root to leaves:
+
+ flt_otel_conf
+ +-- flt_otel_conf_instr (one, via pointer)
+ | +-- flt_otel_conf_ph (ph_groups list)
+ | +-- flt_otel_conf_ph (ph_scopes list)
+ | +-- struct acl (acls list, HAProxy-owned type)
+ | +-- struct logger (proxy_log.loggers, HAProxy type)
+ +-- flt_otel_conf_group (groups list)
+ | +-- flt_otel_conf_ph (ph_scopes list)
+ +-- flt_otel_conf_scope (scopes list)
+ +-- flt_otel_conf_context (contexts list)
+ +-- flt_otel_conf_span (spans list)
+ | +-- flt_otel_conf_link (links list)
+ | +-- flt_otel_conf_sample (attributes list)
+ | | +-- flt_otel_conf_sample_expr (exprs list)
+ | +-- flt_otel_conf_sample (events list)
+ | | +-- flt_otel_conf_sample_expr (exprs list)
+ | +-- flt_otel_conf_sample (baggages list)
+ | | +-- flt_otel_conf_sample_expr (exprs list)
+ | +-- flt_otel_conf_sample (statuses list)
+ | +-- flt_otel_conf_sample_expr (exprs list)
+ +-- flt_otel_conf_str (spans_to_finish list)
+ +-- flt_otel_conf_instrument (instruments list)
+ | +-- flt_otel_conf_sample (samples list)
+ | +-- flt_otel_conf_sample_expr (exprs list)
+ +-- flt_otel_conf_log_record (log_records list)
+ +-- flt_otel_conf_sample (samples list)
+ +-- flt_otel_conf_sample_expr (exprs list)
+
+All child lists use HAProxy's intrusive doubly-linked list (struct list)
+threaded through the FLT_OTEL_CONF_HDR embedded in each child structure.
+
+3.1 Placeholder Structures
+
+The flt_otel_conf_ph structure serves as an indirection node. During parsing,
+placeholder entries record names of groups and scopes. At check time
+(flt_otel_check), these names are resolved to pointers to the actual
+flt_otel_conf_group or flt_otel_conf_scope structures via the ptr field.
+Two type aliases exist for clarity:
+
+ #define flt_otel_conf_ph_group flt_otel_conf_ph
+ #define flt_otel_conf_ph_scope flt_otel_conf_ph
+
+Corresponding free aliases ensure the FLT_OTEL_LIST_DESTROY macro can locate
+the correct free function:
+
+ #define flt_otel_conf_ph_group_free flt_otel_conf_ph_free
+ #define flt_otel_conf_ph_scope_free flt_otel_conf_ph_free
+
+
+4 Structure Definitions
+------------------------------------------------------------------------------
+
+4.1 flt_otel_conf (root)
+
+ proxy Proxy owning the filter.
+ id The OpenTelemetry filter id.
+ cfg_file The OpenTelemetry filter configuration file name.
+ instr The OpenTelemetry instrumentation settings (pointer).
+ groups List of all available groups.
+ scopes List of all available scopes.
+ cnt Various counters related to filter operation.
+ smp_args Deferred sample fetch arguments to resolve at check time.
+
+This structure does not use FLT_OTEL_CONF_HDR because it is not part of any
+list -- it is the unique root, owned by the filter instance.
+
+4.2 flt_otel_conf_instr (instrumentation)
+
+ FLT_OTEL_CONF_HDR(id) The OpenTelemetry instrumentation name.
+ config The OpenTelemetry configuration file name.
+ tracer The OpenTelemetry tracer handle.
+ meter The OpenTelemetry meter handle.
+ logger The OpenTelemetry logger handle.
+ rate_limit Rate limit as uint32 ([0..2^32-1] maps [0..100]%).
+ flag_harderr Hard-error mode flag.
+ flag_disabled Disabled flag.
+ logging Logging mode (0, 1, or 3).
+ proxy_log The log server list (HAProxy proxy structure).
+ analyzers Defined channel analyzers bitmask.
+ idle_timeout Minimum idle timeout across scopes (ms, 0 = off).
+ acls ACLs declared on this tracer.
+ ph_groups List of all used groups (placeholders).
+ ph_scopes List of all used scopes (placeholders).
+
+Exactly one instrumentation block is allowed per filter instance. The parser
+stores a pointer to it in flt_otel_conf.instr.
+
+4.3 flt_otel_conf_group
+
+ FLT_OTEL_CONF_HDR(id) The group name.
+ flag_used The indication that the group is being used.
+ ph_scopes List of all used scopes (placeholders).
+
+Groups bundle scopes for use with the "otel-group" HAProxy action.
+
+4.4 flt_otel_conf_scope
+
+ FLT_OTEL_CONF_HDR(id) The scope name.
+ flag_used The indication that the scope is being used.
+ event FLT_OTEL_EVENT_* identifier.
+ idle_timeout Idle timeout interval in milliseconds (0 = off).
+ acls ACLs declared on this scope.
+ cond ACL condition to meet.
+ contexts Declared contexts.
+ spans Declared spans.
+ spans_to_finish The list of spans scheduled for finishing.
+ instruments The list of metric instruments.
+ log_records The list of log records.
+
+Each scope binds to a single HAProxy analyzer event (or none, if used only
+through groups).
+
+4.5 flt_otel_conf_span
+
+ FLT_OTEL_CONF_HDR(id) The name of the span.
+ FLT_OTEL_CONF_STR(ref_id) The reference name, if used.
+ FLT_OTEL_CONF_STR(ctx_id) The span context name, if used.
+ ctx_flags The type of storage used for the span context.
+ flag_root Whether this is a root span.
+ links The set of linked span names.
+ attributes The set of key:value attributes.
+ events The set of events with key-value attributes.
+ baggages The set of key:value baggage items.
+ statuses Span status code and description.
+
+The ref_id and ctx_id fields use FLT_OTEL_CONF_STR because they are simple name
+strings without list linkage.
+
+4.6 flt_otel_conf_instrument
+
+ FLT_OTEL_CONF_HDR(id) The name of the instrument.
+ idx Meter instrument index: UNSET (-1) before creation,
+ PENDING (-2) while another thread is creating, or >= 0
+ for the actual meter index.
+ type Instrument type (or UPDATE).
+ aggr_type Aggregation type for the view (create only).
+ description Instrument description (create only).
+ unit Instrument unit (create only).
+ samples Sample expressions for the value.
+ bounds Histogram bucket boundaries (create only).
+ bounds_num Number of histogram bucket boundaries.
+ attr Instrument attributes (update only).
+ attr_len Number of instrument attributes.
+ ref Resolved create-form instrument (update only).
+
+Instruments come in two forms: create-form (defines a new metric with type,
+description, unit, and optional histogram bounds) and update-form (references
+an existing instrument via the ref pointer).
+
+4.7 flt_otel_conf_log_record
+
+ FLT_OTEL_CONF_HDR(id) Required by macro; member <id> is not used directly.
+ severity The severity level.
+ event_id Optional event identifier.
+ event_name Optional event name.
+ span Optional span reference.
+ attr Log record attributes.
+ attr_len Number of log record attributes.
+ samples Sample expressions for the body.
+
+Log records are emitted via the OTel logger at the configured severity. The
+optional span reference associates the log record with an open span at runtime.
+Attributes are stored as key-value pairs added via the 'attr' keyword, which
+can be repeated.
+
+4.8 flt_otel_conf_context
+
+ FLT_OTEL_CONF_HDR(id) The name of the context.
+ flags Storage type from which the span context is extracted.
+
+4.9 flt_otel_conf_sample
+
+ FLT_OTEL_CONF_HDR(key) The list containing sample names.
+ fmt_string Combined sample-expression arguments string.
+ extra Optional supplementary data.
+ exprs Used to chain sample expressions.
+ num_exprs Number of defined expressions.
+ lf_expr The log-format expression.
+ lf_used Whether lf_expr is used instead of exprs.
+
+The extra field carries type-specific data: event name strings (OTELC_VALUE_DATA)
+for span events, status code integers (OTELC_VALUE_INT32) for span statuses.
+
+When the sample value argument contains the "%[" sequence, the parser treats
+it as a log-format string: the lf_used flag is set and the compiled result is
+stored in lf_expr, while the exprs list remains empty. At runtime, if lf_used
+is true, the log-format expression is evaluated via build_logline() instead of
+the sample expression list.
+
+4.10 flt_otel_conf_sample_expr
+
+ FLT_OTEL_CONF_HDR(fmt_expr) The original expression format string.
+ expr The sample expression (struct sample_expr).
+
+4.11 Simple Types
+
+ flt_otel_conf_hdr Generic header; used for simple named entries.
+ flt_otel_conf_str String holder (identical to conf_hdr in layout, but the
+ HDR field is named "str" instead of "id"); used for
+ spans_to_finish.
+ flt_otel_conf_link Span link reference; HDR field named "span".
+ flt_otel_conf_ph Placeholder; carries a ptr field resolved at check time.
+
+
+5 Initialization
+------------------------------------------------------------------------------
+
+5.1 Macro-Generated Init Functions
+
+The FLT_OTEL_CONF_FUNC_INIT macro (conf_funcs.h) generates a function with the
+following signature for each configuration type:
+
+ struct flt_otel_conf_<type> *flt_otel_conf_<type>_init(const char *id, int line, struct list *head, char **err);
+
+The generated function performs these steps:
+
+ 1. Validates that <id> is non-NULL and non-empty.
+ 2. Checks the identifier length against FLT_OTEL_ID_MAXLEN (64).
+ 3. If <head> is non-NULL, iterates the list to reject duplicate identifiers
+ (strcmp match).
+ 4. Allocates the structure with OTELC_CALLOC (zeroed memory).
+ 5. Records the configuration line number in cfg_line.
+ 6. Duplicates the identifier string with OTELC_STRDUP.
+ 7. If <head> is non-NULL, appends the structure to the list via LIST_APPEND.
+ 8. Executes any custom initialization body provided as the third macro
+ argument.
+
+If any step fails, the function sets an error message via FLT_OTEL_ERR and
+returns NULL.
+
+5.2 Custom Initialization Bodies
+
+Several structure types require additional setup beyond what the macro template
+provides. The custom init body runs after the base allocation and list
+insertion succeed:
+
+ conf_sample:
+ LIST_INIT for exprs. Calls lf_expr_init for lf_expr.
+
+ conf_span:
+ LIST_INIT for links, attributes, events, baggages, statuses.
+
+ conf_scope:
+ LIST_INIT for acls, contexts, spans, spans_to_finish, instruments,
+ log_records.
+
+ conf_group:
+ LIST_INIT for ph_scopes.
+
+ conf_instrument:
+ Sets idx and type to OTELC_METRIC_INSTRUMENT_UNSET, aggr_type to
+ OTELC_METRIC_AGGREGATION_UNSET. LIST_INIT for samples.
+
+ conf_log_record:
+ LIST_INIT for samples.
+
+ conf_instr:
+ Sets rate_limit to FLT_OTEL_FLOAT_U32(100.0) (100%). Calls init_new_proxy
+ for proxy_log. LIST_INIT for acls, ph_groups, ph_scopes.
+
+Types with no custom body (hdr, str, link, ph, sample_expr, context) rely
+entirely on the zeroed OTELC_CALLOC allocation.
+
+5.3 Extended Sample Initialization
+
+The flt_otel_conf_sample_init_ex function (conf.c) provides a higher-level
+initialization for sample structures:
+
+ 1. Verifies sufficient arguments in the args[] array.
+ 2. Calls flt_otel_conf_sample_init with the sample key.
+ 3. Copies extra data (event name string or status code integer).
+ 4. Concatenates remaining arguments into the fmt_string via
+ flt_otel_args_concat.
+ 5. Counts the number of sample expressions.
+
+This function is used by the parser for span attributes, events, baggages,
+statuses, and instrument samples.
+
+5.4 Top-Level Initialization
+
+The flt_otel_conf_init function (conf.c) is hand-written rather than
+macro-generated because the root structure does not follow the standard header
+pattern:
+
+ 1. Allocates flt_otel_conf with OTELC_CALLOC.
+ 2. Stores the proxy reference.
+ 3. Initializes the groups and scopes lists.
+
+
+6 Deallocation
+------------------------------------------------------------------------------
+
+6.1 Macro-Generated Free Functions
+
+The FLT_OTEL_CONF_FUNC_FREE macro (conf_funcs.h) generates a function with the
+following signature:
+
+ void flt_otel_conf_<type>_free(struct flt_otel_conf_<type> **ptr);
+
+The generated function performs these steps:
+
+ 1. Checks that both <ptr> and <*ptr> are non-NULL.
+ 2. Executes any custom cleanup body provided as the third macro argument.
+ 3. Frees the identifier string with OTELC_SFREE.
+ 4. Removes the structure from its list with FLT_OTEL_LIST_DEL.
+ 5. Frees the structure with OTELC_SFREE_CLEAR and sets <*ptr> to NULL.
+
+6.2 Custom Cleanup Bodies
+
+Custom cleanup runs before the base teardown, allowing child structures to be
+freed while the parent is still valid:
+
+ conf_sample:
+ Frees fmt_string. If extra is OTELC_VALUE_DATA, frees the data pointer.
+ Destroys the exprs list (sample_expr entries). Deinitializes lf_expr via
+ lf_expr_deinit.
+
+ conf_sample_expr:
+ Releases the HAProxy sample expression via release_sample_expr.
+
+ conf_span:
+ Frees ref_id and ctx_id strings.
+ Destroys links, attributes, events, baggages, and statuses lists.
+
+ conf_instrument:
+ Frees description, unit, and bounds. Destroys the samples list.
+ Destroys the attr key-value array via otelc_kv_destroy.
+
+ conf_log_record:
+ Frees event_name and span strings. Destroys the attr key-value array via
+ otelc_kv_destroy. Destroys the samples list.
+
+ conf_scope:
+ Prunes and frees each ACL entry. Frees the ACL condition via free_acl_cond.
+ Destroys contexts, spans, spans_to_finish, instruments, and log_records
+ lists.
+
+ conf_group:
+ Destroys the ph_scopes list.
+
+ conf_instr:
+ Frees the config string. Prunes and frees each ACL entry. Frees each
+ logger entry from proxy_log.loggers. Destroys the ph_groups and ph_scopes
+ lists.
+
+Types with no custom cleanup (hdr, str, link, ph, context) only run the base
+teardown: free the identifier, unlink, free the structure.
+
+6.3 List Destruction
+
+The FLT_OTEL_LIST_DESTROY(type, head) macro (defined in define.h) iterates a
+list and calls flt_otel_conf_<type>_free for each entry. This macro drives the
+recursive teardown from parent to leaf.
+
+6.4 Top-Level Deallocation
+
+The flt_otel_conf_free function (conf.c) is hand-written:
+
+ 1. Frees the id and cfg_file strings.
+ 2. Calls flt_otel_conf_instr_free for the instrumentation.
+ 3. Destroys the groups list (which recursively frees placeholders).
+ 4. Destroys the scopes list (which recursively frees contexts, spans,
+ instruments, log records, and all their children).
+ 5. Frees the root structure and sets the pointer to NULL.
+
+
+7 Summary of Init/Free Pairs
+------------------------------------------------------------------------------
+
+The following table lists all configuration types and their init/free function
+pairs. Types marked "macro" are generated by the FLT_OTEL_CONF_FUNC_(INIT|FREE)
+macros. The HDR field column shows which member name is used for the common
+header.
+
+ Type HDR field Source Custom init body
+ --------------- --------- ------ -------------------------
+ conf (none) manual groups, scopes
+ conf_hdr id macro (none)
+ conf_str str macro (none)
+ conf_link span macro (none)
+ conf_ph id macro (none)
+ conf_sample_expr fmt_expr macro (none)
+ conf_sample key macro exprs, lf_expr
+ conf_sample (ex) key manual extra, fmt_string, exprs
+ conf_context id macro (none)
+ conf_span id macro 5 sub-lists
+ conf_instrument id macro idx, type, samples
+ conf_log_record id macro samples
+ conf_scope id macro 6 sub-lists
+ conf_group id macro ph_scopes
+ conf_instr id macro rate_limit, proxy_log, acls, ph_groups, ph_scopes
--- /dev/null
+ -----------------------------------------
+ HAProxy OTel filter configuration guide
+ Version 1.0
+ ( Last update: 2026-03-18 )
+ -----------------------------------------
+ Author : Miroslav Zagorac
+ Contact : mzagorac at haproxy dot com
+
+
+SUMMARY
+--------
+
+ 1. Overview
+ 2. HAProxy filter declaration
+ 3. OTel configuration file structure
+ 3.1. OTel scope (top-level)
+ 3.2. "otel-instrumentation" section
+ 3.3. "otel-scope" section
+ 3.4. "otel-group" section
+ 4. YAML configuration file
+ 4.1. Exporters
+ 4.2. Samplers
+ 4.3. Processors
+ 4.4. Readers
+ 4.5. Providers
+ 4.6. Signals
+ 5. HAProxy rule integration
+ 6. Complete examples
+ 6.1. Standalone example (sa)
+ 6.2. Frontend / backend example (fe/be)
+ 6.3. Context propagation example (ctx)
+ 6.4. Comparison example (cmp)
+ 6.5. Empty / minimal example (empty)
+
+
+1. Overview
+------------
+
+The OTel filter configuration consists of two files:
+
+ 1) An OTel configuration file (.cfg) that defines the tracing model: scopes,
+ groups, spans, attributes, events, instrumentation and log-records.
+
+ 2) A YAML configuration file (.yml) that configures the OpenTelemetry SDK
+ pipeline: exporters, samplers, processors, readers, providers and signal
+ routing.
+
+The OTel configuration file is referenced from the HAProxy configuration using
+the 'filter opentelemetry' directive. The YAML file is in turn referenced from
+the OTel configuration file using the 'config' keyword inside the
+"otel-instrumentation" section.
+
+
+2. HAProxy filter declaration
+------------------------------
+
+The OTel filter requires the 'insecure-fork-wanted' keyword in the HAProxy
+'global' section. This is necessary because the OpenTelemetry C++ SDK creates
+background threads for data export and batch processing. HAProxy will refuse
+to load the configuration if this keyword is missing.
+
+ global
+ insecure-fork-wanted
+ ...
+
+The filter is activated by adding a filter directive in the HAProxy
+configuration, in a proxy section (frontend / listen / backend):
+
+ frontend my-frontend
+ ...
+ filter opentelemetry [id <id>] config <otel-cfg-file>
+ ...
+
+If no filter id is specified, 'otel-filter' is used as default. The 'config'
+parameter is mandatory and specifies the path to the OTel configuration file.
+
+Example (from test/sa/haproxy.cfg):
+
+ frontend otel-test-sa-frontend
+ bind *:10080
+ default_backend servers-backend
+
+ acl acl-http-status-ok status 100:399
+
+ filter opentelemetry id otel-test-sa config sa/otel.cfg
+
+ http-response otel-group otel-test-sa http_response_group if acl-http-status-ok
+ http-after-response otel-group otel-test-sa http_after_response_group if !acl-http-status-ok
+
+ backend servers-backend
+ server server-1 127.0.0.1:8000
+
+
+3. OTel configuration file structure
+--------------------------------------
+
+The OTel configuration file uses a simple section-based format. It contains
+three types of sections: one "otel-instrumentation" section (mandatory), zero
+or more "otel-scope" sections, and zero or more "otel-group" sections.
+
+
+3.1. OTel scope (top-level)
+-----------------------------
+
+The file is organized into top-level OTel scopes, each identified by a filter
+id enclosed in square brackets. The filter id must match the id specified in
+the HAProxy 'filter opentelemetry' directive.
+
+ [<filter-id>]
+ otel-instrumentation <name>
+ ...
+
+ otel-group <name>
+ ...
+
+ otel-scope <name>
+ ...
+
+Multiple OTel scopes (for different filter instances) can coexist in the same
+file:
+
+ [my-first-filter]
+ otel-instrumentation instr1
+ ...
+
+ [my-second-filter]
+ otel-instrumentation instr2
+ ...
+
+
+3.2. "otel-instrumentation" section
+-------------------------------------
+
+Exactly one "otel-instrumentation" section must be defined per OTel scope.
+It configures the global behavior of the filter and declares which groups
+and scopes are active.
+
+Syntax:
+
+ otel-instrumentation <name>
+
+Keywords (mandatory):
+
+ config <file>
+ Path to the YAML configuration file for the OpenTelemetry SDK.
+
+Keywords (optional):
+
+ acl <aclname> <criterion> [flags] [operator] <value> ...
+ Declare an ACL. See section 7 of the HAProxy Configuration Manual.
+
+ debug-level <value>
+ Set the debug level bitmask (e.g. 0x77f). Only effective when compiled
+ with OTEL_DEBUG=1.
+
+ groups <name> ...
+ Declare one or more "otel-group" sections used by this instrumentation.
+ Can be repeated on multiple lines.
+
+ log global
+ log <addr> [len <len>] [format <fmt>] <facility> [<level> [<minlvl>]]
+ no log
+ Enable per-instance logging.
+
+ option disabled / no option disabled
+ Disable or enable the filter. Default: enabled.
+
+ option dontlog-normal / no option dontlog-normal
+ Suppress logging for normal (successful) operations. Default: disabled.
+
+ option hard-errors / no option hard-errors
+ Stop all filter processing in a stream after the first error. Default:
+ disabled (errors are non-fatal).
+
+ rate-limit <value>
+ Percentage of streams for which the filter is activated. Floating-point
+ value from 0.0 to 100.0. Default: 100.0.
+
+ scopes <name> ...
+ Declare one or more "otel-scope" sections used by this instrumentation.
+ Can be repeated on multiple lines.
+
+
+Example (from test/sa/otel.cfg):
+
+ [otel-test-sa]
+ otel-instrumentation otel-test-instrumentation
+ debug-level 0x77f
+ log localhost:514 local7 debug
+ config sa/otel.yml
+ option dontlog-normal
+ option hard-errors
+ no option disabled
+ rate-limit 100.0
+
+ groups http_response_group
+ groups http_after_response_group
+
+ scopes on_stream_start
+ scopes on_stream_stop
+
+ scopes client_session_start
+ scopes frontend_tcp_request
+ ...
+ scopes server_session_end
+
+
+3.3. "otel-scope" section
+---------------------------
+
+An "otel-scope" section defines the actions that take place when a particular
+event fires or when a group is triggered.
+
+Syntax:
+
+ otel-scope <name>
+
+Supported keywords:
+
+ span <name> [parent <ref>] [link <ref>] [root]
+ Create a new span or reference an already opened one.
+
+ - 'root' marks this span as the trace root (only one per trace).
+ - 'parent <ref>' sets the parent to an existing span or extracted context
+ name.
+ - 'link <ref>' adds an inline link to another span or context. Multiple
+ inline links can be specified within the argument limit.
+ - If no reference is given, the span becomes a root span.
+
+ A span declaration opens a "sub-context" within the scope: the keywords
+ 'link', 'attribute', 'event', 'baggage', 'status' and 'inject' that follow
+ apply to that span until the next 'span' keyword or the end of the scope.
+
+ Examples:
+ span "HAProxy session" root
+ span "Client session" parent "HAProxy session"
+ span "HTTP request" parent "TCP request" link "HAProxy session"
+ span "Client session" parent "otel_ctx_1"
+
+
+ attribute <key> <sample> ...
+ Set an attribute on the currently active span. A single sample preserves
+ its native type; multiple samples are concatenated as a string.
+
+ Examples:
+ attribute "http.method" method
+ attribute "http.url" url
+ attribute "http.version" str("HTTP/") req.ver
+
+
+ event <name> <key> <sample> ...
+ Add a span event (timestamped annotation) to the currently active span.
+ The data type is always string.
+
+ Examples:
+ event "event_ip" "src" src str(":") src_port
+ event "event_be" "be" be_id str(" ") be_name
+
+
+ baggage <key> <sample> ...
+ Set baggage on the currently active span. Baggage propagates to all child
+ spans. The data type is always string.
+
+ Example:
+ baggage "haproxy_id" var(sess.otel.uuid)
+
+
+ status <code> [<sample> ...]
+ Set the span status. Valid codes: ignore (default), unset, ok, error.
+ An optional description follows the code.
+
+ Examples:
+ status "ok"
+ status "error" str("http.status_code: ") status
+
+
+ link <span> ...
+ Add non-hierarchical links to the currently active span. Multiple span
+ names can be specified. Use this keyword for multiple links (the inline
+ 'link' in 'span' is limited to one).
+
+ Example:
+ link "HAProxy session" "Client session"
+
+
+ inject <name-prefix> [use-vars] [use-headers]
+ Inject span context into an HTTP header carrier and/or HAProxy variables.
+ The prefix names the context; the special prefix '-' generates the name
+ automatically. Default storage: use-headers. The 'use-vars' option
+ requires OTEL_USE_VARS=1 at compile time.
+
+ Example:
+ span "HAProxy session" root
+ inject "otel_ctx_1" use-headers use-vars
+
+
+ extract <name-prefix> [use-vars | use-headers]
+ Extract a previously injected span context from an HTTP header or HAProxy
+ variables. The extracted context can then be used as a parent reference
+ in 'span ... parent <name-prefix>'.
+
+ Example:
+ extract "otel_ctx_1" use-vars
+ span "Client session" parent "otel_ctx_1"
+
+
+ finish <name> ...
+ Close one or more spans or span contexts. Special names:
+ '*' - finish all open spans
+ '*req*' - finish all request-channel spans
+ '*res*' - finish all response-channel spans
+
+ Multiple names can be given on one line. A quoted context name after a
+ span name finishes the associated context as well.
+
+ Examples:
+ finish "Frontend TCP request"
+ finish "Client session" "otel_ctx_2"
+ finish *
+
+
+ instrument <type> <name> [aggr <aggregation>] [desc <description>] [unit <unit>] value <sample> [bounds <bounds>]
+ instrument update <name> [attr <key> <value> ...]
+ Create or update a metric instrument.
+
+ Supported types:
+ cnt_int - counter (uint64)
+ hist_int - histogram (uint64)
+ udcnt_int - up-down counter (int64)
+ gauge_int - gauge (int64)
+
+ Supported aggregation types:
+ drop - measurements are discarded
+ histogram - explicit bucket histogram
+ last_value - last recorded value
+ sum - sum of recorded values
+ default - SDK default for the instrument type
+ exp_histogram - base-2 exponential histogram
+
+ An aggregation type can be specified using the 'aggr' keyword. When
+ specified, a metrics view is registered with the given aggregation
+ strategy. If omitted, the SDK default is used.
+
+ For histogram instruments (hist_int), optional bucket boundaries can be
+ specified using the 'bounds' keyword followed by a double-quoted string
+ of space-separated numbers (order does not matter; values are sorted
+ internally). When bounds are specified without an explicit aggregation
+ type, histogram aggregation is used automatically.
+
+ Observable (asynchronous) and double-precision types are not supported.
+ Observable instrument callbacks are invoked by the OTel SDK from an
+ external background thread; HAProxy sample fetches rely on internal
+ per-thread-group state and return incorrect results from a non-HAProxy
+ thread. Double-precision types are not supported because HAProxy sample
+ fetches do not return double values.
+
+ Examples:
+ instrument cnt_int "name_cnt_int" desc "Integer Counter" value int(1),add(2) unit "unit"
+ instrument hist_int "name_hist" aggr exp_histogram desc "Latency" value lat_ns_tot unit "ns"
+ instrument hist_int "name_hist2" desc "Latency" value lat_ns_tot unit "ns" bounds "100 1000 10000"
+ instrument update "name_cnt_int" attr "attr_1_key" "attr_1_value"
+
+
+ log-record <severity> [id <integer>] [event <name>] [span <span-name>] [attr <key> <value>] ... <sample> ...
+ Emit an OpenTelemetry log record. The first argument is a required
+ severity level. Optional keywords follow in any order:
+
+ id <integer> - numeric event identifier
+ event <name> - event name string
+ span <span-name> - associate the log record with an open span
+ attr <key> <value> - add a key-value attribute (repeatable)
+
+ The remaining arguments at the end are sample fetch expressions that form
+ the log record body. A single sample preserves its native type; multiple
+ samples are concatenated as a string.
+
+ Supported severity levels follow the OpenTelemetry specification:
+ trace, trace2, trace3, trace4
+ debug, debug2, debug3, debug4
+ info, info2, info3, info4
+ warn, warn2, warn3, warn4
+ error, error2, error3, error4
+ fatal, fatal2, fatal3, fatal4
+
+ The log record is only emitted if the logger is enabled for the configured
+ severity (controlled by the 'min_severity' option in the YAML logs signal
+ configuration). If a 'span' reference is given but the named span is not
+ found at runtime, the log record is emitted without span correlation.
+
+ Examples:
+ log-record info str("heartbeat")
+ log-record info id 1001 event "http-request" span "Frontend HTTP request" attr "http.method" "GET" method url
+ log-record trace id 1000 event "session-start" span "Client session" attr "attr_1_key" "attr_1_value" attr "attr_2_key" "attr_2_value" src str(":") src_port
+ log-record warn event "server-unavailable" str("503 Service Unavailable")
+ log-record info event "session-stop" str("stream stopped")
+
+
+ acl <aclname> <criterion> [flags] [operator] <value> ...
+ Declare an ACL local to this scope.
+
+ Example:
+ acl acl-test-src-ip src 127.0.0.1
+
+
+ otel-event <name> [{ if | unless } <condition>]
+ Bind this scope to a filter event, optionally with an ACL-based condition.
+
+ Supported events (stream lifecycle):
+ on-stream-start
+ on-stream-stop
+ on-idle-timeout
+ on-backend-set
+
+ Supported events (request channel):
+ on-client-session-start
+ on-frontend-tcp-request
+ on-http-wait-request
+ on-http-body-request
+ on-frontend-http-request
+ on-switching-rules-request
+ on-backend-tcp-request
+ on-backend-http-request
+ on-process-server-rules-request
+ on-http-process-request
+ on-tcp-rdp-cookie-request
+ on-process-sticking-rules-request
+ on-http-headers-request
+ on-http-end-request
+ on-client-session-end
+ on-server-unavailable
+
+ Supported events (response channel):
+ on-server-session-start
+ on-tcp-response
+ on-http-wait-response
+ on-process-store-rules-response
+ on-http-response
+ on-http-headers-response
+ on-http-end-response
+ on-http-reply
+ on-server-session-end
+
+ The on-stream-start event fires from the stream_start filter callback,
+ before any channel processing begins. The on-stream-stop event fires from
+ the stream_stop callback, after all channel processing ends. No channel
+ is available at that point, so context injection/extraction via HTTP
+ headers cannot be used in scopes bound to these events.
+
+ The on-idle-timeout event fires periodically when the stream has no data
+ transfer activity. It requires the 'idle-timeout' keyword to set the
+ interval. Scopes bound to this event can create heartbeat spans, record
+ idle-time metrics, and emit idle-time log records.
+
+ The on-backend-set event fires when a backend is assigned to the stream.
+ It is not called if the frontend and backend are the same.
+
+ The on-http-headers-request and on-http-headers-response events fire after
+ all HTTP headers have been parsed and analyzed.
+
+ The on-http-end-request and on-http-end-response events fire when all HTTP
+ data has been processed and forwarded.
+
+ The on-http-reply event fires when HAProxy generates an internal reply
+ (error page, deny response, redirect).
+
+ Examples:
+ otel-event on-stream-start if acl-test-src-ip
+ otel-event on-stream-stop
+ otel-event on-client-session-start
+ otel-event on-client-session-start if acl-test-src-ip
+ otel-event on-http-response if !acl-http-status-ok
+ otel-event on-idle-timeout
+
+
+ idle-timeout <time>
+ Set the idle timeout interval for a scope bound to the 'on-idle-timeout'
+ event. The timer fires periodically at the given interval when the stream
+ has no data transfer activity. This keyword is mandatory for scopes using
+ the 'on-idle-timeout' event and cannot be used with any other event.
+
+ The <time> argument accepts the standard HAProxy time format: a number
+ followed by a unit suffix (ms, s, m, h, d). A value of zero is not
+ permitted.
+
+ Example:
+ scopes on_idle_timeout
+ ..
+ otel-scope on_idle_timeout
+ idle-timeout 5s
+ span "heartbeat" root
+ attribute "idle.elapsed" str("idle-check")
+ instrument cnt_int "idle.count" value int(1)
+ log-record info str("heartbeat")
+ otel-event on-idle-timeout
+
+
+3.4. "otel-group" section
+---------------------------
+
+An "otel-group" section defines a named collection of scopes that can be
+triggered from HAProxy TCP/HTTP rules rather than from filter events.
+
+Syntax:
+
+ otel-group <name>
+
+Keywords:
+
+ scopes <name> ...
+ List the "otel-scope" sections that belong to this group. Multiple names
+ can be given on one line. Scopes that are used only in groups do not need
+ to define an 'otel-event'.
+
+Example (from test/sa/otel.cfg):
+
+ otel-group http_response_group
+ scopes http_response_1
+ scopes http_response_2
+
+ otel-scope http_response_1
+ span "HTTP response"
+ event "event_content" "hdr.content" res.hdr("content-type") str("; length: ") res.hdr("content-length") str(" bytes")
+
+ otel-scope http_response_2
+ span "HTTP response"
+ event "event_date" "hdr.date" res.hdr("date") str(" / ") res.hdr("last-modified")
+
+
+4. YAML configuration file
+----------------------------
+
+The YAML configuration file defines the OpenTelemetry SDK pipeline. It is
+referenced by the 'config' keyword in the "otel-instrumentation" section.
+It contains the following top-level sections: exporters, samplers, processors,
+readers, providers and signals.
+
+
+4.1. Exporters
+---------------
+
+Each exporter has a user-chosen name and a 'type' that determines which
+additional options are available. Options marked with (*) are required.
+
+Supported types:
+
+ otlp_grpc - Export via OTLP over gRPC.
+ type (*) "otlp_grpc"
+ thread_name exporter thread name (string)
+ endpoint OTLP/gRPC endpoint URL (string)
+ use_ssl_credentials enable SSL channel credentials (boolean)
+ ssl_credentials_cacert_path CA certificate file path (string)
+ ssl_credentials_cacert_as_string CA certificate as inline string (string)
+ ssl_client_key_path client private key file path (string)
+ ssl_client_key_string client private key as inline string (string)
+ ssl_client_cert_path client certificate file path (string)
+ ssl_client_cert_string client certificate as inline string (string)
+ timeout export timeout in seconds (integer)
+ user_agent User-Agent header value (string)
+ max_threads maximum exporter threads (integer)
+ compression compression algorithm name (string)
+ max_concurrent_requests concurrent request limit (integer)
+
+
+ otlp_http - Export via OTLP over HTTP (JSON or Protobuf).
+ type (*) "otlp_http"
+ thread_name exporter thread name (string)
+ endpoint OTLP/HTTP endpoint URL (string)
+ content_type payload format: "json" or "protobuf"
+ json_bytes_mapping binary encoding: "hexid", "utf8" or "base64"
+ debug enable debug output (boolean)
+ timeout export timeout in seconds (integer)
+ http_headers custom HTTP headers (list of key: value)
+ max_concurrent_requests concurrent request limit (integer)
+ max_requests_per_connection request limit per connection (integer)
+ ssl_insecure_skip_verify skip TLS certificate verification (boolean)
+ ssl_ca_cert_path CA certificate file path (string)
+ ssl_ca_cert_string CA certificate as inline string (string)
+ ssl_client_key_path client private key file path (string)
+ ssl_client_key_string client private key as inline string (string)
+ ssl_client_cert_path client certificate file path (string)
+ ssl_client_cert_string client certificate as inline string (string)
+ ssl_min_tls minimum TLS version (string)
+ ssl_max_tls maximum TLS version (string)
+ ssl_cipher TLS cipher list (string)
+ ssl_cipher_suite TLS 1.3 cipher suite list (string)
+ compression compression algorithm name (string)
+
+
+ otlp_file - Export to local files in OTLP format.
+ type (*) "otlp_file"
+ thread_name exporter thread name (string)
+ file_pattern output filename pattern (string)
+ alias_pattern symlink pattern for latest file (string)
+ flush_interval flush interval in microseconds (integer)
+ flush_count spans per flush (integer)
+ file_size maximum file size in bytes (integer)
+ rotate_size number of rotated files to keep (integer)
+
+
+ ostream - Write to a file (text output, useful for debugging).
+ type (*) "ostream"
+ filename output file path (string)
+
+
+ memory - In-memory buffer (useful for testing).
+ type (*) "memory"
+ buffer_size maximum buffered items (integer)
+
+
+ zipkin - Export to Zipkin-compatible backends.
+ type (*) "zipkin"
+ endpoint Zipkin collector URL (string)
+ format payload format: "json" or "protobuf"
+ service_name service name reported to Zipkin (string)
+ ipv4 service IPv4 address (string)
+ ipv6 service IPv6 address (string)
+
+
+ elasticsearch - Export to Elasticsearch.
+ type (*) "elasticsearch"
+ host Elasticsearch hostname (string)
+ port Elasticsearch port (integer)
+ index Elasticsearch index name (string)
+ response_timeout response timeout in seconds (integer)
+ debug enable debug output (boolean)
+ http_headers custom HTTP headers (list of key: value)
+
+
+4.2. Samplers
+--------------
+
+Samplers control which traces are recorded. Each sampler has a user-chosen
+name and a 'type' that determines its behavior.
+
+Supported types:
+
+ always_on - Sample every trace.
+ type (*) "always_on"
+
+
+ always_off - Sample no traces.
+ type (*) "always_off"
+
+
+ trace_id_ratio_based - Sample a fraction of traces.
+ type (*) "trace_id_ratio_based"
+ ratio sampling ratio, 0.0 to 1.0 (float)
+
+
+ parent_based - Inherit sampling decision from parent span.
+ type (*) "parent_based"
+ delegate fallback sampler name (string)
+
+
+4.3. Processors
+----------------
+
+Processors define how telemetry data is handled before export. Each
+processor has a user-chosen name and a 'type' that determines its behavior.
+
+Supported types:
+
+ batch - Batch spans before exporting.
+ type (*) "batch"
+ thread_name processor thread name (string)
+ max_queue_size maximum queued spans (integer)
+ schedule_delay export interval in milliseconds (integer)
+ max_export_batch_size maximum spans per export call (integer)
+
+ When the queue reaches half capacity, a preemptive notification triggers
+ an early export.
+
+ single - Export each span individually (no batching).
+ type (*) "single"
+
+
+4.4. Readers
+-------------
+
+Readers define how metrics are collected and exported. Each reader has a
+user-chosen name.
+
+ thread_name reader thread name (string)
+ export_interval collection interval in milliseconds (integer)
+ export_timeout export timeout in milliseconds (integer)
+
+
+4.5. Providers
+---------------
+
+Providers define resource attributes attached to all telemetry data. Each
+provider has a user-chosen name.
+
+ resources key-value resource attributes (list)
+
+Standard resource attribute keys include service.name, service.version,
+service.instance.id and service.namespace.
+
+
+4.6. Signals
+-------------
+
+Signals bind exporters, samplers, processors, readers and providers together
+for each telemetry type. The supported signal names are "traces", "metrics"
+and "logs".
+
+ scope_name instrumentation scope name (string)
+ exporters exporter name reference (string)
+ samplers sampler name reference (string, traces only)
+ processors processor name reference (string, traces/logs)
+ readers reader name reference (string, metrics only)
+ providers provider name reference (string)
+ min_severity minimum log severity level (string, logs only)
+
+The "min_severity" option controls which log records are emitted. Only log
+records whose severity is equal to or higher than the configured minimum are
+passed to the exporter. The value is a severity name as listed under the
+"log-record" keyword (e.g. "trace", "debug", "info", "warn", "error", "fatal").
+If omitted, the logger accepts all severity levels.
+
+
+5. HAProxy rule integration
+----------------------------
+
+Groups defined in the OTel configuration file can be triggered from HAProxy
+TCP/HTTP rules using the 'otel-group' action keyword:
+
+ http-request otel-group <filter-id> <group> [condition]
+ http-response otel-group <filter-id> <group> [condition]
+ http-after-response otel-group <filter-id> <group> [condition]
+ tcp-request otel-group <filter-id> <group> [condition]
+ tcp-response otel-group <filter-id> <group> [condition]
+
+This allows running specific groups of scopes based on ACL conditions defined
+in the HAProxy configuration.
+
+Example (from test/sa/haproxy.cfg):
+
+ acl acl-http-status-ok status 100:399
+
+ filter opentelemetry id otel-test-sa config sa/otel.cfg
+
+ # Run response scopes for successful responses
+ http-response otel-group otel-test-sa http_response_group if acl-http-status-ok
+
+ # Run after-response scopes for error responses
+ http-after-response otel-group otel-test-sa http_after_response_group if !acl-http-status-ok
+
+
+6. Complete examples
+---------------------
+
+The test directory contains several complete example configurations. Each
+subdirectory contains an OTel configuration file (otel.cfg), a YAML file
+(otel.yml) and a HAProxy configuration file (haproxy.cfg).
+
+
+6.1. Standalone example (sa)
+------------------------------
+
+The most comprehensive example. All possible events are used, with spans,
+attributes, events, links, baggage, status, metrics and groups demonstrated.
+
+--- test/sa/otel.cfg (excerpt) -----------------------------------------
+
+[otel-test-sa]
+ otel-instrumentation otel-test-instrumentation
+ config sa/otel.yml
+ option dontlog-normal
+ option hard-errors
+ no option disabled
+ rate-limit 100.0
+
+ groups http_response_group
+ groups http_after_response_group
+
+ scopes on_stream_start
+ scopes on_stream_stop
+ scopes client_session_start
+ scopes frontend_tcp_request
+ ...
+ scopes server_session_end
+
+ otel-group http_response_group
+ scopes http_response_1
+ scopes http_response_2
+
+ otel-scope http_response_1
+ span "HTTP response"
+ event "event_content" "hdr.content" res.hdr("content-type") str("; length: ") res.hdr("content-length") str(" bytes")
+
+ otel-scope on_stream_start
+ instrument udcnt_int "haproxy.sessions.active" desc "Active sessions" value int(1) unit "{session}"
+ span "HAProxy session" root
+ baggage "haproxy_id" var(sess.otel.uuid)
+ event "event_ip" "src" src str(":") src_port
+ acl acl-test-src-ip src 127.0.0.1
+ otel-event on-stream-start if acl-test-src-ip
+
+ otel-scope on_stream_stop
+ finish *
+ otel-event on-stream-stop
+
+ otel-scope client_session_start
+ span "Client session" parent "HAProxy session"
+ otel-event on-client-session-start
+
+ otel-scope frontend_http_request
+ span "Frontend HTTP request" parent "HTTP body request" link "HAProxy session"
+ attribute "http.method" method
+ attribute "http.url" url
+ attribute "http.version" str("HTTP/") req.ver
+ finish "HTTP body request"
+ otel-event on-frontend-http-request
+
+ otel-scope server_session_start
+ span "Server session" parent "HAProxy session"
+ link "HAProxy session" "Client session"
+ finish "Process sticking rules request"
+ otel-event on-server-session-start
+
+ otel-scope server_session_end
+ finish *
+ otel-event on-server-session-end
+
+---------------------------------------------------------------------
+
+
+6.2. Frontend / backend example (fe/be)
+-----------------------------------------
+
+Demonstrates distributed tracing across two cascaded HAProxy instances using
+inject/extract to propagate the span context via HTTP headers.
+
+The frontend HAProxy (test/fe) creates the root trace and injects context:
+
+--- test/fe/otel.cfg (excerpt) -----------------------------------------
+
+ otel-scope backend_http_request
+ span "Backend HTTP request" parent "Backend TCP request"
+ finish "Backend TCP request"
+ span "HAProxy session"
+ inject "otel-ctx" use-headers
+ otel-event on-backend-http-request
+
+---------------------------------------------------------------------
+
+The backend HAProxy (test/be) extracts the context and continues the trace:
+
+--- test/be/otel.cfg (excerpt) -----------------------------------------
+
+ otel-scope frontend_http_request
+ extract "otel-ctx" use-headers
+ span "HAProxy session" parent "otel-ctx" root
+ baggage "haproxy_id" var(sess.otel.uuid)
+ span "Client session" parent "HAProxy session"
+ span "Frontend HTTP request" parent "Client session"
+ attribute "http.method" method
+ attribute "http.url" url
+ attribute "http.version" str("HTTP/") req.ver
+ otel-event on-frontend-http-request
+
+---------------------------------------------------------------------
+
+
+6.3. Context propagation example (ctx)
+----------------------------------------
+
+Similar to 'sa', but spans are opened using extracted span contexts as parent
+references instead of direct span names. This demonstrates the inject/extract
+mechanism using HAProxy variables.
+
+--- test/ctx/otel.cfg (excerpt) ----------------------------------------
+
+ otel-scope client_session_start_1
+ span "HAProxy session" root
+ inject "otel_ctx_1" use-headers use-vars
+ baggage "haproxy_id" var(sess.otel.uuid)
+ otel-event on-client-session-start
+
+ otel-scope client_session_start_2
+ extract "otel_ctx_1" use-vars
+ span "Client session" parent "otel_ctx_1"
+ inject "otel_ctx_2" use-headers use-vars
+ otel-event on-client-session-start
+
+ otel-scope frontend_tcp_request
+ extract "otel_ctx_2" use-vars
+ span "Frontend TCP request" parent "otel_ctx_2"
+ inject "otel_ctx_3" use-headers use-vars
+ otel-event on-frontend-tcp-request
+
+ otel-scope http_wait_request
+ extract "otel_ctx_3" use-vars
+ span "HTTP wait request" parent "otel_ctx_3"
+ finish "Frontend TCP request" "otel_ctx_3"
+ otel-event on-http-wait-request
+
+---------------------------------------------------------------------
+
+
+6.4. Comparison example (cmp)
+-------------------------------
+
+A configuration made for comparison purposes with other tracing implementations.
+It uses a simplified span hierarchy without context propagation.
+
+--- test/cmp/otel.cfg (excerpt) ----------------------------------------
+
+ otel-scope client_session_start
+ span "HAProxy session" root
+ baggage "haproxy_id" var(sess.otel.uuid)
+ span "Client session" parent "HAProxy session"
+ otel-event on-client-session-start
+
+ otel-scope http_response-error
+ span "HTTP response"
+ status "error" str("!acl-http-status-ok")
+ otel-event on-http-response if !acl-http-status-ok
+
+ otel-scope server_session_end
+ finish "HTTP response" "Server session"
+ otel-event on-http-response
+
+ otel-scope client_session_end
+ finish "*"
+ otel-event on-http-response
+
+---------------------------------------------------------------------
+
+
+6.5. Empty / minimal example (empty)
+--------------------------------------
+
+The minimal valid OTel configuration. The filter is initialized but no events
+are triggered:
+
+--- test/empty/otel.cfg -------------------------------------------------
+
+ otel-instrumentation otel-test-instrumentation
+ config empty/otel.yml
+
+---------------------------------------------------------------------
+
+This is useful for testing the OTel filter initialization behavior without any
+actual telemetry processing.
--- /dev/null
+OpenTelemetry filter -- function reference
+==========================================================================
+
+Functions are grouped by source file. Functions marked with [D] are only
+compiled when DEBUG_OTEL is defined.
+
+
+src/filter.c
+----------------------------------------------------------------------
+
+Filter lifecycle callbacks and helpers registered in flt_otel_ops.
+
+ flt_otel_mem_malloc
+ Allocator callback for the OTel C wrapper library. Uses the HAProxy
+ pool_head_otel_span_context pool.
+
+ flt_otel_mem_free
+ Deallocator callback for the OTel C wrapper library.
+
+ flt_otel_log_handler_cb
+ Diagnostic callback for the OTel C wrapper library. Counts SDK internal
+ diagnostic messages.
+
+ flt_otel_thread_id
+ Returns the current HAProxy thread ID (tid).
+
+ flt_otel_lib_init
+ Initializes the OTel C wrapper library: verifies the library version,
+ constructs the configuration path, calls otelc_init(), and creates the
+ tracer, meter and logger instances.
+
+ flt_otel_is_disabled
+ Checks whether the filter instance is disabled for the current stream.
+ Logs the event name when DEBUG_OTEL is enabled.
+
+ flt_otel_return_int
+ Error handler for callbacks returning int. In hard-error mode, disables
+ the filter; in soft-error mode, clears the error and returns OK.
+
+ flt_otel_return_void
+ Error handler for callbacks returning void. Same logic as
+ flt_otel_return_int but without a return value.
+
+ flt_otel_ops_init
+ Filter init callback (flt_ops.init). Called once per proxy to initialize
+ the OTel library via flt_otel_lib_init() and register CLI keywords.
+
+ flt_otel_ops_deinit
+ Filter deinit callback (flt_ops.deinit). Destroys the tracer, meter and
+ logger, frees the configuration, and calls otelc_deinit().
+
+ flt_otel_ops_check
+ Filter check callback (flt_ops.check). Validates the parsed
+ configuration: checks for duplicate filter IDs, resolves group/scope
+ placeholder references, verifies root span count, and sets analyzer bits.
+
+ flt_otel_ops_init_per_thread
+ Per-thread init callback (flt_ops.init_per_thread). Starts the OTel
+ tracer thread and enables HTX filtering.
+
+ flt_otel_ops_deinit_per_thread [D]
+ Per-thread deinit callback (flt_ops.deinit_per_thread).
+
+ flt_otel_ops_attach
+ Filter attach callback (flt_ops.attach). Called when a filter instance is
+ attached to a stream. Applies rate limiting, creates the runtime context,
+ and sets analyzer bits.
+
+ flt_otel_ops_stream_start
+ Stream start callback (flt_ops.stream_start). Fires the
+ on-stream-start event before any channel processing begins. The channel
+ argument is NULL. After the event, initializes the idle timer in the
+ runtime context from the precomputed minimum idle_timeout in the
+ instrumentation configuration.
+
+ flt_otel_ops_stream_set_backend
+ Stream set-backend callback (flt_ops.stream_set_backend). Fires the
+ on-backend-set event when a backend is assigned to the stream.
+
+ flt_otel_ops_stream_stop
+ Stream stop callback (flt_ops.stream_stop). Fires the
+ on-stream-stop event after all channel processing ends. The channel
+ argument is NULL.
+
+ flt_otel_ops_detach
+ Filter detach callback (flt_ops.detach). Frees the runtime context when
+ the filter is detached from a stream.
+
+ flt_otel_ops_check_timeouts
+ Timeout callback (flt_ops.check_timeouts). When the idle-timeout timer
+ has expired, fires the on-idle-timeout event and reschedules the timer
+ for the next interval. Sets the STRM_EVT_MSG pending event flag on the
+ stream.
+
+ flt_otel_ops_channel_start_analyze
+ Channel start-analyze callback. Registers analyzers on the channel and
+ runs the client/server session start event. Propagates the idle-timeout
+ expiry to the channel's analyse_exp so the stream task keeps waking.
+
+ flt_otel_ops_channel_pre_analyze
+ Channel pre-analyze callback. Maps the analyzer bit to an event index and
+ runs the corresponding event.
+
+ flt_otel_ops_channel_post_analyze
+ Channel post-analyze callback. Non-resumable; called once when a
+ filterable analyzer finishes.
+
+ flt_otel_ops_channel_end_analyze
+ Channel end-analyze callback. Runs the client/server session end event.
+ For the request channel, also fires the server-unavailable event if no
+ response was processed.
+
+ flt_otel_ops_http_headers
+ HTTP headers callback (flt_ops.http_headers). Fires
+ on-http-headers-request or on-http-headers-response depending on the
+ channel direction.
+
+ flt_otel_ops_http_payload [D]
+ HTTP payload callback (flt_ops.http_payload).
+
+ flt_otel_ops_http_end
+ HTTP end callback (flt_ops.http_end). Fires on-http-end-request or
+ on-http-end-response depending on the channel direction.
+
+ flt_otel_ops_http_reset [D]
+ HTTP reset callback (flt_ops.http_reset).
+
+ flt_otel_ops_http_reply
+ HTTP reply callback (flt_ops.http_reply). Fires the on-http-reply event
+ when HAProxy generates an internal reply.
+
+ flt_otel_ops_tcp_payload [D]
+ TCP payload callback (flt_ops.tcp_payload).
+
+
+src/event.c
+----------------------------------------------------------------------
+
+Event dispatching, metrics recording and scope/span execution engine.
+
+ flt_otel_scope_run_instrument_record
+ Records a measurement for a synchronous metric instrument. Evaluates the
+ sample expression from the create-form instrument (instr_ref) and submits
+ the value to the meter via update_instrument_kv_n(), using per-scope
+ attributes from the update-form instrument (instr).
+
+ flt_otel_scope_run_instrument
+ Processes all metric instruments for a scope. Runs in two passes: the
+ first lazily creates create-form instruments via the meter, using
+ HA_ATOMIC_CAS to guarantee thread-safe one-time initialization; the second
+ iterates update-form instruments and records measurements via
+ flt_otel_scope_run_instrument_record(). Instruments whose index is still
+ negative (UNUSED or PENDING) are skipped.
+
+ flt_otel_scope_run_log_record
+ Emits log records for a scope. Iterates over the configured log-record
+ list, skipping entries whose severity is below the logger threshold.
+ Evaluates the body from sample fetch expressions or a log-format string,
+ optionally resolves a span reference against the runtime context, and
+ emits the record via the logger. A missing span is non-fatal -- the
+ record is emitted without span correlation.
+
+ flt_otel_scope_run_span
+ Executes a single span: creates the OTel span on first call, adds links,
+ baggage, attributes, events and status, then injects the context into HTTP
+ headers or HAProxy variables.
+
+ flt_otel_scope_run
+ Executes a complete scope: evaluates ACL conditions, extracts contexts,
+ iterates over configured spans (resolving links, evaluating sample
+ expressions), calls flt_otel_scope_run_span for each, processes metric
+ instruments via flt_otel_scope_run_instrument(), emits log records via
+ flt_otel_scope_run_log_record(), then marks and finishes completed spans.
+
+ flt_otel_event_run
+ Top-level event dispatcher. Called from filter callbacks, iterates over
+ all scopes matching the event index and calls flt_otel_scope_run() for
+ each.
+
+
+src/scope.c
+----------------------------------------------------------------------
+
+Runtime context, span and context lifecycle management.
+
+ flt_otel_pools_info [D]
+ Logs the sizes of all registered HAProxy memory pools used by the OTel
+ filter.
+
+ flt_otel_runtime_context_init
+ Allocates and initializes the per-stream runtime context. Generates a
+ UUID and stores it in the sess.otel.uuid HAProxy variable.
+
+ flt_otel_runtime_context_free
+ Frees the runtime context: ends all active spans, destroys all extracted
+ contexts, and releases pool memory.
+
+ flt_otel_scope_span_init
+ Finds an existing scope span by name or creates a new one. Resolves the
+ parent reference (span or extracted context).
+
+ flt_otel_scope_span_free
+ Frees a scope span entry if its OTel span has been ended. Refuses to free
+ an active (non-NULL) span.
+
+ flt_otel_scope_context_init
+ Finds an existing scope context by name or creates a new one by extracting
+ the span context from a text map.
+
+ flt_otel_scope_context_free
+ Frees a scope context entry and destroys the underlying OTel span context.
+
+ flt_otel_scope_data_dump [D]
+ Dumps scope data contents (baggage, attributes, events, links, status) for
+ debugging.
+
+ flt_otel_scope_data_init
+ Zero-initializes a scope data structure and its event/link lists.
+
+ flt_otel_scope_data_free
+ Frees all scope data contents: key-value arrays, event entries, link
+ entries, and status description.
+
+ flt_otel_scope_finish_mark
+ Marks spans and contexts for finishing. Supports wildcard ("*"),
+ channel-specific ("req"/"res"), and named targets.
+
+ flt_otel_scope_finish_marked
+ Ends all spans and destroys all contexts that have been marked for
+ finishing by flt_otel_scope_finish_mark().
+
+ flt_otel_scope_free_unused
+ Removes scope spans with NULL OTel span and scope contexts with NULL OTel
+ context. Cleans up associated HTTP headers and variables.
+
+
+src/parser.c
+----------------------------------------------------------------------
+
+Configuration file parsing for otel-instrumentation, otel-group and otel-scope
+sections.
+
+ flt_otel_parse_strdup
+ Duplicates a string with error handling; optionally stores the string
+ length.
+
+ flt_otel_parse_keyword
+ Parses a single keyword argument: checks for duplicates and missing
+ values, then stores via flt_otel_parse_strdup().
+
+ flt_otel_parse_invalid_char
+ Validates characters in a name according to the specified type
+ (identifier, domain, context prefix, variable).
+
+ flt_otel_parse_cfg_check
+ Common validation for config keywords: looks up the keyword, checks
+ argument count and character validity, verifies that the parent section ID
+ is set.
+
+ flt_otel_parse_cfg_sample_expr
+ Parses a single HAProxy sample expression within a sample definition.
+ Calls sample_parse_expr().
+
+ flt_otel_parse_cfg_sample
+ Parses a complete sample definition (key plus one or more sample
+ expressions).
+
+ flt_otel_parse_cfg_str
+ Parses one or more string arguments into a conf_str list (used for the
+ "finish" keyword).
+
+ flt_otel_parse_cfg_file
+ Parses and validates a file path argument; checks that the file exists and
+ is readable.
+
+ flt_otel_parse_check_scope
+ Checks whether the current config parsing is within the correct HAProxy
+ configuration scope (cfg_scope filtering).
+
+ flt_otel_parse_cfg_instr
+ Section parser for the otel-instrumentation block. Handles keywords:
+ otel-instrumentation ID, log, config, groups, scopes, acl, rate-limit,
+ option, debug-level.
+
+ flt_otel_post_parse_cfg_instr
+ Post-parse callback for otel-instrumentation. Links the instrumentation
+ to the config and checks that a config file is specified.
+
+ flt_otel_parse_cfg_group
+ Section parser for the otel-group block. Handles keywords: otel-group ID,
+ scopes.
+
+ flt_otel_post_parse_cfg_group
+ Post-parse callback for otel-group. Checks that at least one scope is
+ defined.
+
+ flt_otel_parse_cfg_scope_ctx
+ Parses the context storage type argument ("use-headers" or "use-vars") for
+ inject/extract keywords.
+
+ flt_otel_parse_acl
+ Builds an ACL condition by trying multiple ACL lists in order
+ (scope-local, instrumentation, proxy).
+
+ flt_otel_parse_bounds
+ Parses a space-separated string of numbers into a dynamically allocated
+ array of doubles for histogram bucket boundaries. Sorts the values
+ internally.
+
+ flt_otel_parse_cfg_instrument
+ Parses the "instrument" keyword inside an otel-scope section. Supports
+ both "update" form (referencing an existing instrument) and "create" form
+ (defining a new metric instrument with type, name, optional aggregation
+ type, description, unit, value, and optional histogram bounds).
+
+ flt_otel_parse_cfg_scope
+ Section parser for the otel-scope block. Handles keywords: otel-scope ID,
+ span, link, attribute, event, baggage, status, inject, extract, finish,
+ instrument, log-record, acl, otel-event.
+
+ flt_otel_post_parse_cfg_scope
+ Post-parse callback for otel-scope. Checks that HTTP header injection is
+ only used on events that support it.
+
+ flt_otel_parse_cfg
+ Parses the OTel filter configuration file. Backs up current sections,
+ registers temporary otel-instrumentation/group/scope section parsers,
+ loads and parses the file, then restores the original sections.
+
+ flt_otel_parse
+ Main filter parser entry point, registered for the "otel" filter keyword.
+ Parses the filter ID and configuration file path from the HAProxy config
+ line.
+
+
+src/conf.c
+----------------------------------------------------------------------
+
+Configuration structure allocation and deallocation. Most init/free pairs are
+generated by the FLT_OTEL_CONF_FUNC_INIT and FLT_OTEL_CONF_FUNC_FREE macros.
+
+ flt_otel_conf_hdr_init
+ Allocates and initializes a conf_hdr structure.
+
+ flt_otel_conf_hdr_free
+ Frees a conf_hdr structure and removes it from its list.
+
+ flt_otel_conf_str_init
+ Allocates and initializes a conf_str structure.
+
+ flt_otel_conf_str_free
+ Frees a conf_str structure and removes it from its list.
+
+ flt_otel_conf_link_init
+ Allocates and initializes a conf_link structure (span link).
+
+ flt_otel_conf_link_free
+ Frees a conf_link structure and removes it from its list.
+
+ flt_otel_conf_ph_init
+ Allocates and initializes a conf_ph (placeholder) structure.
+
+ flt_otel_conf_ph_free
+ Frees a conf_ph structure and removes it from its list.
+
+ flt_otel_conf_sample_expr_init
+ Allocates and initializes a conf_sample_expr structure.
+
+ flt_otel_conf_sample_expr_free
+ Frees a conf_sample_expr structure and releases the parsed sample
+ expression.
+
+ flt_otel_conf_sample_init
+ Allocates and initializes a conf_sample structure.
+
+ flt_otel_conf_sample_init_ex
+ Extended sample initialization: sets the key, extra data (event name or
+ status code), concatenated value string, and expression count.
+
+ flt_otel_conf_sample_free
+ Frees a conf_sample structure including its value, extra data, and all
+ sample expressions.
+
+ flt_otel_conf_context_init
+ Allocates and initializes a conf_context structure.
+
+ flt_otel_conf_context_free
+ Frees a conf_context structure and removes it from its list.
+
+ flt_otel_conf_span_init
+ Allocates and initializes a conf_span structure with empty lists for
+ links, attributes, events, baggages and statuses.
+
+ flt_otel_conf_span_free
+ Frees a conf_span structure and all its child lists.
+
+ flt_otel_conf_instrument_init
+ Allocates and initializes a conf_instrument structure.
+
+ flt_otel_conf_instrument_free
+ Frees a conf_instrument structure and removes it from its list.
+
+ flt_otel_conf_log_record_init
+ Allocates and initializes a conf_log_record structure with an empty
+ samples list.
+
+ flt_otel_conf_log_record_free
+ Frees a conf_log_record structure: event_name, span, attributes and
+ samples list.
+
+ flt_otel_conf_scope_init
+ Allocates and initializes a conf_scope structure with empty lists for
+ ACLs, contexts, spans, spans_to_finish and instruments.
+
+ flt_otel_conf_scope_free
+ Frees a conf_scope structure, ACLs, condition, and all child lists.
+
+ flt_otel_conf_group_init
+ Allocates and initializes a conf_group structure with an empty placeholder
+ scope list.
+
+ flt_otel_conf_group_free
+ Frees a conf_group structure and its placeholder scope list.
+
+ flt_otel_conf_instr_init
+ Allocates and initializes a conf_instr structure. Sets the default rate
+ limit to 100%, initializes the proxy_log, and creates empty ACL and
+ placeholder lists.
+
+ flt_otel_conf_instr_free
+ Frees a conf_instr structure including ACLs, loggers, config path, and
+ placeholder lists.
+
+ flt_otel_conf_init
+ Allocates and initializes the top-level flt_otel_conf structure with empty
+ group and scope lists.
+
+ flt_otel_conf_free
+ Frees the top-level flt_otel_conf structure and all of its children
+ (instrumentation, groups, scopes).
+
+
+src/cli.c
+----------------------------------------------------------------------
+
+HAProxy CLI command handlers for runtime filter management.
+
+ cmn_cli_set_msg
+ Sets the CLI appctx response message and state.
+
+ flt_otel_cli_parse_debug [D]
+ CLI handler for "otel debug [level]". Gets or sets the debug level.
+
+ flt_otel_cli_parse_disabled
+ CLI handler for "otel enable" and "otel disable".
+
+ flt_otel_cli_parse_option
+ CLI handler for "otel soft-errors" and "otel hard-errors".
+
+ flt_otel_cli_parse_logging
+ CLI handler for "otel logging [state]". Gets or sets the logging state
+ (off/on/nolognorm).
+
+ flt_otel_cli_parse_rate
+ CLI handler for "otel rate [value]". Gets or sets the rate limit
+ percentage.
+
+ flt_otel_cli_parse_status
+ CLI handler for "otel status". Displays filter configuration and runtime
+ state for all OTel filter instances.
+
+ flt_otel_cli_init
+ Registers the OTel CLI keywords with HAProxy.
+
+
+src/otelc.c
+----------------------------------------------------------------------
+
+OpenTelemetry context propagation bridge (inject/extract) between HAProxy and
+the OTel C wrapper library.
+
+ flt_otel_text_map_writer_set_cb
+ Writer callback for text map injection. Appends a key-value pair to the
+ text map.
+
+ flt_otel_http_headers_writer_set_cb
+ Writer callback for HTTP headers injection. Appends a key-value pair to
+ the text map.
+
+ flt_otel_inject_text_map
+ Injects span context into a text map carrier.
+
+ flt_otel_inject_http_headers
+ Injects span context into an HTTP headers carrier.
+
+ flt_otel_text_map_reader_foreach_key_cb
+ Reader callback for text map extraction. Iterates over all key-value
+ pairs in the text map.
+
+ flt_otel_http_headers_reader_foreach_key_cb
+ Reader callback for HTTP headers extraction. Iterates over all key-value
+ pairs in the text map.
+
+ flt_otel_extract_text_map
+ Extracts a span context from a text map carrier via the tracer.
+
+ flt_otel_extract_http_headers
+ Extracts a span context from an HTTP headers carrier via the tracer.
+
+
+src/http.c
+----------------------------------------------------------------------
+
+HTTP header manipulation for context propagation.
+
+ flt_otel_http_headers_dump [D]
+ Dumps all HTTP headers from the channel's HTX buffer.
+
+ flt_otel_http_headers_get
+ Extracts HTTP headers matching a prefix into a text map. Used by the
+ "extract" keyword to read span context from incoming request headers.
+
+ flt_otel_http_header_set
+ Sets or removes an HTTP header. Combines prefix and name into the full
+ header name, removes all existing occurrences, then adds the new value
+ (if non-NULL).
+
+ flt_otel_http_headers_remove
+ Removes all HTTP headers matching a prefix. Wrapper around
+ flt_otel_http_header_set() with NULL name and value.
+
+
+src/vars.c
+----------------------------------------------------------------------
+
+HAProxy variable integration for context propagation and storage. Only compiled
+when USE_OTEL_VARS is defined.
+
+ flt_otel_vars_scope_dump [D]
+ Dumps all variables for a single HAProxy variable scope.
+
+ flt_otel_vars_dump [D]
+ Dumps all variables across all scopes (PROC, SESS, TXN, REQ/RES).
+
+ flt_otel_smp_init
+ Initializes a sample structure with stream ownership and optional string
+ data.
+
+ flt_otel_smp_add
+ Appends a context variable name to the binary sample data buffer used for
+ tracking registered context variables.
+
+ flt_otel_normalize_name
+ Normalizes a variable name: replaces dashes with 'D' and spaces with 'S',
+ converts to lowercase.
+
+ flt_otel_denormalize_name
+ Reverses the normalization applied by flt_otel_normalize_name(). Restores
+ dashes from 'D' and spaces from 'S'.
+
+ flt_otel_var_name
+ Constructs a full variable name from scope, prefix and name components,
+ separated by dots.
+
+ flt_otel_ctx_loop
+ Iterates over all context variable names stored in the binary sample data,
+ calling a callback for each.
+
+ flt_otel_ctx_set_cb
+ Callback for flt_otel_ctx_loop() that checks whether a context variable
+ name already exists.
+
+ flt_otel_ctx_set
+ Registers a context variable name in the binary tracking buffer if it is
+ not already present.
+
+ flt_otel_var_register
+ Registers a HAProxy variable via vars_check_arg() so it can be used at
+ runtime.
+
+ flt_otel_var_set
+ Sets a HAProxy variable value. For context-scope variables, also
+ registers the name in the context tracking buffer.
+
+ flt_otel_vars_unset_cb
+ Callback for flt_otel_ctx_loop() that unsets each context variable.
+
+ flt_otel_vars_unset
+ Unsets all context variables for a given prefix and removes the tracking
+ variable itself.
+
+ flt_otel_vars_get_scope
+ Resolves a scope name string ("proc", "sess", "txn", "req", "res") to the
+ corresponding HAProxy variable store.
+
+ flt_otel_vars_get_cb
+ Callback for flt_otel_ctx_loop() that reads each context variable value
+ and adds it to a text map.
+
+ flt_otel_vars_get
+ Reads all context variables for a prefix into a text map. Used by the
+ "extract" keyword with variable storage.
+
+
+src/pool.c
+----------------------------------------------------------------------
+
+Memory pool and trash buffer helpers.
+
+ flt_otel_pool_alloc
+ Allocates memory from a HAProxy pool (if available) or from the heap.
+ Optionally zero-fills the allocated block.
+
+ flt_otel_pool_strndup
+ Duplicates a string using a HAProxy pool (if available) or the heap.
+
+ flt_otel_pool_free
+ Returns memory to a HAProxy pool or frees it from the heap.
+
+ flt_otel_trash_alloc
+ Allocates a trash buffer chunk, optionally zero-filled.
+
+ flt_otel_trash_free
+ Frees a trash buffer chunk.
+
+
+src/util.c
+----------------------------------------------------------------------
+
+Utility and conversion functions.
+
+ flt_otel_args_dump [D]
+ Dumps configuration arguments array to stderr.
+
+ flt_otel_filters_dump [D]
+ Dumps all OTel filter instances across all proxies.
+
+ flt_otel_chn_label [D]
+ Returns "REQuest" or "RESponse" based on channel flags.
+
+ flt_otel_pr_mode [D]
+ Returns "HTTP" or "TCP" based on proxy mode.
+
+ flt_otel_stream_pos [D]
+ Returns "frontend" or "backend" based on stream flags.
+
+ flt_otel_type [D]
+ Returns "frontend" or "backend" based on filter flags.
+
+ flt_otel_analyzer [D]
+ Returns the analyzer name string for a given analyzer bit.
+
+ flt_otel_list_dump [D]
+ Returns a summary string for a list (empty, single, count).
+
+ flt_otel_args_count
+ Counts the number of valid (non-NULL) arguments in an args array, handling
+ gaps from blank arguments.
+
+ flt_otel_args_concat
+ Concatenates arguments starting from a given index into a single
+ space-separated string.
+
+ flt_otel_strtod
+ Parses a string to double with range validation.
+
+ flt_otel_strtoll
+ Parses a string to int64 with range validation.
+
+ flt_otel_sample_to_str
+ Converts sample data to its string representation. Handles bool, sint,
+ IPv4, IPv6, str, and HTTP method types.
+
+ flt_otel_sample_to_value
+ Converts sample data to an otelc_value. Preserves native types (bool,
+ int64) where possible; falls back to string.
+
+ flt_otel_sample_add_event
+ Adds a sample value as a span event attribute. Groups attributes by event
+ name; dynamically grows the attribute array.
+
+ flt_otel_sample_set_status
+ Sets the span status code and description from sample data.
+
+ flt_otel_sample_add_kv
+ Adds a sample value as a key-value attribute or baggage entry.
+ Dynamically grows the key-value array.
+
+ flt_otel_sample_add
+ Top-level sample evaluator. Processes all sample expressions for a
+ configured sample, converts results, and dispatches to the appropriate
+ handler (attribute, event, baggage, status).
+
+
+src/group.c
+----------------------------------------------------------------------
+
+Group action support for http-response / http-after-response / tcp-request /
+tcp-response rules.
+
+ flt_otel_group_action
+ Action callback (action_ptr) for the otel-group rule. Finds the filter
+ instance on the current stream and runs all scopes defined in the group.
+
+ flt_otel_group_check
+ Check callback (check_ptr) for the otel-group rule. Resolves filter ID
+ and group ID references against the proxy's filter configuration.
+
+ flt_otel_group_release
+ Release callback (release_ptr) for the otel-group rule.
+
+ flt_otel_group_parse
+ Parses the "otel-group" action keyword from HAProxy config rules.
+ Registered for tcp-request, tcp-response, http-request, http-response and
+ http-after-response action contexts.
--- /dev/null
+OpenTelemetry Filter Implementation Review
+======================================================================
+
+1 Overview
+----------------------------------------------------------------------
+
+The OpenTelemetry (OTel) filter for HAProxy provides distributed tracing,
+metrics and logging capabilities. It creates, propagates and exports spans,
+metric instruments and log records that follow the OpenTelemetry specification.
+The filter hooks into the HAProxy stream processing pipeline through the
+filter API and maps HAProxy channel analyzer events to OpenTelemetry span
+lifecycle operations, metric recordings and log-record emissions.
+
+The implementation is located entirely under addons/otel/ and consists of
+header files, C source files, a Makefile, and a set of test configurations
+with runner scripts.
+
+
+2 Directory Structure
+----------------------------------------------------------------------
+
+ addons/otel/
+ |-- Makefile Build integration (USE_OTEL option)
+ |-- include/
+ | |-- include.h Master include (pulls all headers)
+ | |-- config.h Build-time tunables (pool sizes, limits)
+ | |-- define.h Utility macros (memory, strings, lists)
+ | |-- debug.h Debug/logging infrastructure
+ | |-- filter.h Filter return codes, alert macros
+ | |-- parser.h Configuration keyword definitions
+ | |-- conf.h Configuration data structures
+ | |-- conf_funcs.h Generated init/free function macros
+ | |-- event.h Event enumeration and data table
+ | |-- scope.h Runtime span/context structures
+ | |-- pool.h Memory pool helpers
+ | |-- http.h HTTP header manipulation
+ | |-- otelc.h Span context inject/extract wrappers
+ | |-- vars.h HAProxy variable integration
+ | |-- util.h String conversion, sample helpers
+ | |-- group.h Group action (HAProxy rule integration)
+ | `-- cli.h CLI command interface
+ |-- src/
+ | |-- filter.c Filter lifecycle and channel callbacks
+ | |-- parser.c Configuration file parser
+ | |-- conf.c Configuration structure init/free
+ | |-- event.c Scope/span execution engine
+ | |-- scope.c Runtime context and span management
+ | |-- http.c HTTP header get/set/remove
+ | |-- otelc.c C wrapper inject/extract bridge
+ | |-- vars.c HAProxy variable read/write
+ | |-- pool.c Pool alloc/free, trash buffers
+ | |-- util.c Argument handling, sample conversion
+ | |-- group.c Group action parsing and execution
+ | `-- cli.c CLI command handlers
+ `-- test/
+ |-- copy-yml.sh YAML configuration transformer
+ |-- test-speed.sh Performance benchmarking runner
+ |-- run-sa.sh Standalone test runner
+ |-- run-fe-be.sh Frontend-backend chain runner
+ |-- run-ctx.sh Context propagation test runner
+ |-- run-cmp.sh Comparison test runner
+ |-- sa/ Standalone test configs
+ |-- fe/ Frontend-only test configs
+ |-- be/ Backend-only test configs
+ |-- ctx/ Context propagation test configs
+ |-- cmp/ Comparison test configs
+ `-- empty/ Minimal/empty configuration test
+
+
+3 Build System
+----------------------------------------------------------------------
+
+The Makefile is included from the main HAProxy build when USE_OTEL is set.
+It detects the opentelemetry-c-wrapper library via pkg-config or manual
+OTEL_INC/OTEL_LIB paths.
+
+Build options:
+
+ USE_OTEL=1 Enable the filter (required).
+ OTEL_DEBUG=1 Compile with DEBUG_OTEL; links the _dbg variant of the
+ wrapper library and enables additional debug callbacks in
+ filter.c (stream_set_backend, http_headers, http_payload,
+ tcp_payload, etc.).
+ OTEL_USE_VARS=1 Compile vars.c; enables USE_OTEL_VARS which allows span
+ context propagation via HAProxy transaction variables in
+ addition to HTTP headers.
+ OTEL_INC=<path> Manual include path for the C wrapper.
+ OTEL_LIB=<path> Manual library path for the C wrapper.
+ OTEL_RUNPATH=1 Embed RPATH to the wrapper library.
+
+Compiled objects (11 always, 12 with OTEL_USE_VARS):
+
+ cli.o conf.o event.o filter.o group.o http.o opentelemetry.o parser.o
+ pool.o scope.o util.o [vars.o]
+
+
+4 Configuration Parsing
+----------------------------------------------------------------------
+
+Configuration parsing is driven by parser.c. The filter is declared in the
+HAProxy configuration with:
+
+ filter opentelemetry [id <name>] config <file>
+
+The flt_otel_parse() function (parser.c) handles the "filter" line, creates an
+flt_otel_conf structure, and delegates to parse_cfg() which loads the referenced
+YAML/CFG file. That file is parsed using temporary section registrations for
+three section types:
+
+ otel-instrumentation -> flt_otel_parse_cfg_instr()
+ otel-group -> flt_otel_parse_cfg_group()
+ otel-scope -> flt_otel_parse_cfg_scope()
+
+After each section is fully parsed, a post-parse function validates the section
+(e.g., flt_otel_post_parse_cfg_scope() checks that context injection is only
+used on events that support it).
+
+4.1 Instrumentation Section
+
+ otel-instrumentation <name>
+ config <file>
+ log <target>
+ debug-level <value>
+ rate-limit <value>
+ option { disabled | hard-errors | dontlog-normal }
+ groups <name> ...
+ scopes <name> ...
+ acl <name> <criterion> ...
+
+The instrumentation block defines global filter parameters: the YAML exporter
+configuration file, logging, rate limiting, and references to groups and scopes.
+Exactly one instrumentation block is allowed per filter instance.
+
+4.2 Group Section
+
+ otel-group <name>
+ scopes <name> ...
+
+Groups bundle multiple scopes under a single name for use with HAProxy
+http-request/http-response rules via the "otel-group" action. The group action
+(group.c) parses the rule, resolves the scope references at check time, and
+executes all referenced scopes when the rule fires.
+
+4.3 Scope Section
+
+ otel-scope <name>
+ otel-event <event-name> [if|unless <condition>]
+ extract <name-prefix> [use-headers|use-vars]
+ span <name> [parent <ref>] [link <ref>] [root]
+ link <span> ...
+ attribute <key> <sample> ...
+ event <name> <key> <sample> ...
+ baggage <key> <sample> ...
+ status <code> [<sample> ...]
+ inject <name-prefix> [use-headers] [use-vars]
+ finish <name> ...
+ instrument <type> <name> ... / instrument update <name> ...
+ log-record <severity> [id <int>] [event <name>] [span <ref>] [attr <k> <v>] ... <sample> ...
+ acl <name> <criterion> ...
+
+Each scope ties to a single HAProxy analyzer event (or none, if used only
+through groups). Scopes contain context extraction directives, span
+definitions, metric instruments, log records, and finish directives.
+
+A span may specify:
+ - A parent reference (another span or extracted context).
+ - One or more links to other spans/contexts. Inline link syntax allows one
+ link on the span line; the standalone "link" keyword allows multiple.
+ - The "root" flag marking it as the trace root.
+ - Attributes, events, baggages and status evaluated from HAProxy sample
+ expressions at runtime.
+ - An inject directive to propagate the span context via HTTP headers and/or
+ HAProxy variables.
+
+4.4 Configuration Structure Initialization
+
+All configuration structures are allocated and freed using macro-generated
+functions from conf_funcs.h:
+
+ FLT_OTEL_CONF_FUNC_INIT(type, id_field, extra_init)
+ FLT_OTEL_CONF_FUNC_FREE(type, id_field, extra_free)
+
+These macros produce flt_otel_conf_<type>_init() and _free() functions.
+The init function:
+ - Checks the identifier length against FLT_OTEL_ID_MAXLEN (64).
+ - Checks for duplicate identifiers in the target list.
+ - Allocates the structure with OTELC_CALLOC.
+ - Copies the identifier with OTELC_STRDUP.
+ - Appends to the head list.
+ - Executes any extra initialization (e.g., LIST_INIT for sub-lists in the
+ span structure).
+
+The free function:
+ - Executes any extra cleanup (e.g., destroying sub-lists).
+ - Frees the identifier string.
+ - Removes the node from its list.
+ - Frees the structure.
+
+The full init/free chain for all structures:
+
+ flt_otel_conf flt_otel_conf_init() / flt_otel_conf_free()
+ flt_otel_conf_instr generated via macro
+ flt_otel_conf_ph generated (for ph_groups, ph_scopes)
+ flt_otel_conf_group generated
+ flt_otel_conf_ph generated (for ph_scopes)
+ flt_otel_conf_scope generated
+ flt_otel_conf_context generated
+ flt_otel_conf_span generated
+ flt_otel_conf_link generated
+ flt_otel_conf_sample generated + _init_ex()
+ flt_otel_conf_sample_expr generated
+ flt_otel_conf_instrument generated
+ flt_otel_conf_log_record generated
+ flt_otel_conf_sample generated + _init_ex()
+ flt_otel_conf_sample_expr generated
+
+
+5 Filter Lifecycle
+----------------------------------------------------------------------
+
+The filter registers its operations in the flt_otel_ops structure (filter.c)
+and the keyword parser via INITCALL1 (parser.c).
+
+5.1 Proxy-Level Initialization
+
+ flt_otel_ops_init():
+ - Registers CLI commands via flt_otel_cli_init().
+ - Initializes the OpenTelemetry library via flt_otel_lib_init(): verifies
+ the C wrapper version, resolves the absolute path of the YAML
+ configuration file, calls otelc_init() to set up exporters, creates the
+ tracer, meter and logger objects, and registers custom memory allocation
+ and thread-id callbacks with the wrapper via otelc_ext_init().
+
+ flt_otel_ops_check():
+ - Validates that filter IDs are unique across all proxies.
+ - Resolves group->scope and instrumentation->scope/group placeholder
+ references to actual configuration structures (setting the ptr field
+ and flag_used).
+ - Warns about unused scopes, missing root spans, or multiple root spans.
+ - Validates metric instruments: resolves update-form references to their
+ matching create-form definitions, and rejects duplicate create-form names
+ across scopes.
+ - Computes the aggregated analyzer bitmask from all used scopes.
+
+ flt_otel_ops_init_per_thread():
+ - Starts the tracer, meter and logger background threads on first call.
+ - Sets the FLT_CFG_FL_HTX flag to enable HTX stream filtering.
+
+ flt_otel_ops_deinit():
+ - Destroys the tracer, meter and logger.
+ - Frees the entire configuration tree.
+ - Calls otelc_deinit() to shut down the wrapper library.
+
+5.2 Stream-Level Callbacks
+
+ flt_otel_ops_attach():
+ - Checks if the filter is globally disabled; returns IGNORE.
+ - Applies rate limiting via ha_random32(); returns IGNORE if the random
+ value exceeds the configured rate_limit.
+ - Creates the runtime context (flt_otel_runtime_context_init) with a
+ generated UUID and initialized span/context lists.
+ - Sets pre_analyzers and post_analyzers bitmasks from the instrumentation's
+ aggregated analyzer flags. AN_REQ_WAIT_HTTP and AN_RES_WAIT_HTTP are
+ placed in post_analyzers because those analyzers can only be used in the
+ post_analyze callback. AN_REQ_HTTP_TARPIT is excluded from pre_analyzers.
+
+ flt_otel_ops_detach():
+ - Frees the runtime context, which finishes all remaining active spans and
+ destroys all remaining contexts.
+
+ flt_otel_ops_check_timeouts():
+ - Checks whether the idle-timeout timer has expired; if so, fires the
+ on-idle-timeout event and reschedules the timer for the next interval.
+ - Sets STRM_EVT_MSG on the stream's pending_events to ensure the filter is
+ re-evaluated after a timeout.
+
+5.3 Error Handling
+
+Two helper functions manage errors:
+
+ flt_otel_return_int() / flt_otel_return_void():
+ - If the result indicates an error or an error string is set: in hard-error
+ mode, the filter is disabled for the current stream (flag_disabled = 1)
+ and the disabled counter is incremented atomically. In soft-error mode,
+ the error is merely logged.
+ - The error string is always freed.
+ - For int returns, FLT_OTEL_RET_OK is returned regardless, so the stream
+ continues processing even after an error.
+
+
+6 Event Processing (Channel Analyzers)
+----------------------------------------------------------------------
+
+The filter maps HAProxy channel analyzer callbacks to a table of named events
+defined in event.h (FLT_OTEL_EVENT_DEFINES).
+
+6.1 Event Table
+
+Each event entry carries:
+ - an_bit: the HAProxy analyzer bit (AN_REQ_*, AN_RES_*)
+ - an_name: the analyzer bit name (e.g. "AN_REQ_FLT_HTTP_HDRS")
+ - smp_opt_dir: sample fetch direction (REQ or RES)
+ - smp_val_fe/be: valid sample fetch locations
+ - flag_http_inject: whether span context can be injected into HTTP headers
+ at this point
+ - name: configuration event name (e.g. "on-frontend-http-request")
+
+Events with an_bit == 0 are pseudo-events not tied to any channel
+analyzer. Two of them fire from stream lifecycle callbacks:
+ - on-stream-start (flt_otel_ops_stream_start, before channel processing)
+ - on-stream-stop (flt_otel_ops_stream_stop, after channel processing)
+
+One fires periodically from the check_timeouts callback:
+ - on-idle-timeout (flt_otel_ops_check_timeouts, when stream is idle)
+
+One fires from the stream_set_backend callback:
+ - on-backend-set (flt_otel_ops_stream_set_backend, when backend is assigned)
+
+Four fire from HTTP lifecycle callbacks:
+ - on-http-headers-request / on-http-headers-response (flt_otel_ops_http_headers)
+ - on-http-end-request / on-http-end-response (flt_otel_ops_http_end)
+ - on-http-reply (flt_otel_ops_http_reply)
+
+The remaining pseudo-events fire from channel start/end callbacks:
+ - on-client-session-start / on-client-session-end
+ - on-server-session-start / on-server-session-end
+ - on-server-unavailable
+
+The stream lifecycle events pass NULL for the channel argument, so
+context injection/extraction via HTTP headers cannot be used. Their
+sample fetch direction is unconstrained (0xff), allowing both request
+and response fetches.
+
+6.2 Callback Flow
+
+ stream_start(s, f):
+ - Fires on-stream-start with chn=NULL.
+ - Called when a new stream begins, before any channel processing.
+ - Initializes the idle timer from the precomputed minimum idle_timeout in
+ the instrumentation configuration.
+
+ stream_set_backend(s, f, be):
+ - Fires on-backend-set with chn=&s->req.
+ - Called when a backend is assigned (skipped if frontend == backend).
+
+ stream_stop(s, f):
+ - Fires on-stream-stop with chn=NULL.
+ - Called when a stream is destroyed, after all channel processing.
+
+ check_timeouts(s, f):
+ - Checks whether the idle-timeout timer has expired.
+ - If expired, fires on-idle-timeout and reschedules the timer.
+
+ channel_start_analyze(chn):
+ - Enables the per-channel analyzers from pre_analyzers.
+ - Fires on-client-session-start (request) or on-server-session-start
+ (response).
+ - Propagates the idle-timeout expiry to the channel's analyse_exp.
+
+ channel_pre_analyze(chn, an_bit):
+ - Looks up the event by an_bit in the event table.
+ - Calls flt_otel_event_run() for the matching event.
+
+ channel_post_analyze(chn, an_bit):
+ - Same as pre_analyze but for post-analyzers (AN_REQ_WAIT_HTTP,
+ AN_RES_WAIT_HTTP).
+
+ channel_end_analyze(chn):
+ - Fires on-client-session-end (request) or on-server-session-end (response).
+ - For the request channel: if response analyzers were configured but
+ none executed (server was unreachable), fires on-server-unavailable.
+
+ http_headers(s, f, msg):
+ - Fires on-http-headers-request or on-http-headers-response depending on
+ msg->chn direction.
+
+ http_end(s, f, msg):
+ - Fires on-http-end-request or on-http-end-response depending on
+ msg->chn direction.
+
+ http_reply(s, f, status, msg):
+ - Fires on-http-reply with chn=&s->res.
+
+6.3 Scope Execution
+
+ flt_otel_event_run() (event.c):
+ - Captures timestamps (CLOCK_MONOTONIC + CLOCK_REALTIME).
+ - Updates the runtime context's executed-analyzers bitmask.
+ - Iterates all scopes matching the event; calls flt_otel_scope_run() for
+ each used scope.
+
+ flt_otel_scope_run() (event.c):
+ 1. Evaluates the scope's ACL condition; if it fails:
+ - If the scope contains a root span, disables the stream.
+ - Returns without processing.
+ 2. Extracts contexts: for each configured extract directive, reads
+ the span context from HTTP headers or HAProxy variables via
+ flt_otel_scope_context_init().
+ 3. Processes spans: for each configured span:
+ a. Calls flt_otel_scope_span_init() which either returns an existing
+ scope_span (by name) or creates a new one with resolved parent
+ reference.
+ b. Resolves span links against the runtime context -- first searching
+ active spans, then extracted contexts. Unresolved links produce a
+ NOTICE-level warning and are skipped.
+ c. Evaluates attributes, events, baggages, and status from sample
+ expressions via flt_otel_sample_add().
+ d. Calls flt_otel_scope_run_span() which:
+ - Creates the OTel span via tracer->start_span_with_options()
+ (if not already started).
+ - Adds all resolved links via span->add_link().
+ - Sets baggage, attributes, events, and status.
+ - Optionally injects the span context into HTTP headers and/or
+ HAProxy variables.
+ 4. Processes metric instruments via flt_otel_scope_run_instrument(), which
+ runs two passes: the first lazily creates create-form instruments using
+ HA_ATOMIC_CAS for thread-safe one-time initialization; the second records
+ measurements for update-form instruments, skipping any whose index is
+ still negative (creation pending or not yet attempted).
+ 5. Emits log records via flt_otel_scope_run_log_record(), which iterates
+ the scope's log-record list, skips entries below the logger's severity
+ threshold, evaluates sample expressions into a body string, resolves
+ the optional span reference, and emits the record via the logger.
+ 6. Marks spans listed in "finish" directives.
+ 7. Calls flt_otel_scope_finish_marked() to end marked spans/contexts.
+ 8. Calls flt_otel_scope_free_unused() to remove finished and destroyed
+ scope_span/scope_context entries from the runtime lists.
+
+
+7 Runtime Data Structures
+----------------------------------------------------------------------
+
+7.1 Runtime Context (per stream)
+
+ flt_otel_runtime_context:
+ stream Owning stream pointer.
+ filter Owning filter pointer.
+ uuid[40] Generated UUID v4 for the session.
+ flag_harderr Copied from instrumentation config.
+ flag_disabled Set when the filter encounters a hard error or ACL disables
+ processing.
+ logging Logging flags.
+ analyzers Bitmask of analyzers that have actually executed.
+ idle_timeout Idle timeout interval in milliseconds (0 = off).
+ idle_exp Tick at which the next idle timeout fires.
+ spans Linked list of flt_otel_scope_span.
+ contexts Linked list of flt_otel_scope_context.
+
+7.2 Scope Span
+
+ flt_otel_scope_span:
+ id / id_len Span operation name (borrowed from config).
+ smp_opt_dir Direction in which the span was created.
+ flag_finish Set by finish directives, cleared after ending.
+ span The OTel span object (NULL before start, NULL after
+ end_with_options).
+ ref_span Parent span pointer (resolved at init).
+ ref_ctx Parent context pointer (resolved at init).
+ list Chain in runtime_context.spans.
+
+ flt_otel_scope_span_init() performs memoization: if a span with the same name
+ already exists in rt_ctx->spans, it returns the existing entry. This allows
+ multiple scopes to contribute attributes/events to the same logical span.
+
+7.3 Scope Context
+
+ flt_otel_scope_context:
+ id / id_len Context name (borrowed from config).
+ smp_opt_dir Direction in which the context was extracted.
+ flag_finish Marks the context for destruction.
+ context The OTel span_context object.
+ list Chain in runtime_context.contexts.
+
+ Similarly memoized: duplicate extraction of the same context name returns the
+ existing entry.
+
+7.4 Scope Data (per span per scope run, stack-allocated)
+
+ flt_otel_scope_data:
+ baggage Key-value array for baggage items.
+ attributes Key-value array for span attributes.
+ events Linked list of flt_otel_scope_data_event (each with name
+ + key-value array).
+ links Linked list of flt_otel_scope_data_link (each with span
+ and/or context pointer).
+ status Status code and description string.
+
+ Initialized at the start of each span processing block and freed at the end.
+ The link entries hold borrowed pointers to OTel objects owned by the runtime
+ context, so only the link nodes themselves are freed.
+
+7.5 Span Finishing
+
+ finish <name> / finish * / finish *req* / finish *res*
+
+ The "finish" directive marks spans and contexts for completion:
+ - "*" marks all.
+ - "*req*" / "*res*" marks those created in the request/response direction
+ respectively.
+ - Otherwise, marks by exact name.
+
+ flt_otel_scope_finish_marked() iterates all marked entries:
+ - Spans are ended via span->end_with_options() which NULLs the span pointer.
+ - Contexts are destroyed via context->destroy() which NULLs the context
+ pointer.
+
+ flt_otel_scope_free_unused() then removes entries with NULL span/context
+ pointers from the runtime lists. For contexts, associated HTTP headers
+ and variables are also cleaned up.
+
+ On stream detach (flt_otel_runtime_context_free), any remaining active spans
+ are force-ended and all entries are freed.
+
+
+8 Span Links
+----------------------------------------------------------------------
+
+Span links associate a span with other spans or contexts without establishing
+a parent-child relationship.
+
+8.1 Configuration
+
+Two syntaxes are supported:
+
+ Inline (one link per span declaration):
+ span <name> [parent <ref>] link <linked-span> [root]
+
+ Standalone (multiple links, requires a preceding span):
+ link <span-name> [<span-name> ...]
+
+The flt_otel_conf_link structure stores each link target name. Duplicate link
+names within the same span are rejected by the init macro's duplicate check.
+The links list is initialized in flt_otel_conf_span_init() and destroyed in
+flt_otel_conf_span_free().
+
+8.2 Runtime Resolution
+
+At scope execution time (event.c, flt_otel_scope_run), for each configured link:
+ 1. The name is searched in rt_ctx->spans (active scope_span entries).
+ If found, the OTel span pointer is captured.
+ 2. If not found in spans, the name is searched in rt_ctx->contexts (extracted
+ scope_context entries). If found, the OTel span_context pointer is
+ captured.
+ 3. If neither found, a NOTICE warning is logged and the link is skipped.
+ 4. A flt_otel_scope_data_link node is allocated and appended to the scope
+ data's links list.
+
+In flt_otel_scope_run_span(), all resolved links are applied via
+span->add_link(span, link_span, link_context, NULL, 0). The last two arguments
+(attributes array and count) are NULL/0, meaning links carry no additional
+attributes.
+
+
+9 Context Propagation
+----------------------------------------------------------------------
+
+9.1 Extraction
+
+ extract <name-prefix> [use-headers|use-vars]
+
+Extracts an incoming trace context. The prefix identifies the header name
+pattern (for HTTP) or variable name pattern (for vars).
+
+ - use-headers (default): flt_otel_http_headers_get() iterates HTX headers
+ matching the prefix and builds an otelc_text_map.
+ - use-vars: flt_otel_vars_get() reads HAProxy variables matching the prefix
+ pattern.
+
+The text map is passed to flt_otel_extract_http_headers() which uses the
+C wrapper to reconstruct an otelc_span_context.
+
+9.2 Injection
+
+ inject <name-prefix> [use-headers] [use-vars]
+
+Injects the current span's context into outgoing data. Both storage types can
+be used simultaneously.
+
+ flt_otel_inject_http_headers() serializes the span context into an
+ otelc_http_headers_writer which produces a text_map. For each key-value pair:
+ - use-headers: flt_otel_http_header_set() adds/replaces the header with the
+ prefixed name.
+ - use-vars: flt_otel_var_register() + flt_otel_var_set() stores the value
+ in a HAProxy transaction variable with normalized name (dashes replaced
+ with 'D', spaces with 'S', uppercase lowered; dots serve as component
+ separators).
+
+
+10 HTTP Header Manipulation
+----------------------------------------------------------------------
+
+ http.c provides three operations:
+
+ flt_otel_http_headers_get(chn, prefix, prefix_len, err):
+ Iterates the HTX message headers. Headers whose name starts with the given
+ prefix are collected into an otelc_text_map. The prefix is stripped from
+ the names in the returned map.
+
+ flt_otel_http_header_set(chn, prefix, name, value, err):
+ Removes any existing header matching "prefix" + "name", then adds a new
+ header with the given value. If name is NULL, all headers with the prefix
+ are removed (bulk delete).
+
+ flt_otel_http_headers_remove(chn, prefix, err):
+ Convenience wrapper; removes all headers matching the prefix.
+
+
+11 HAProxy Variable Integration
+----------------------------------------------------------------------
+
+Enabled with OTEL_USE_VARS=1. Provides an alternative propagation mechanism
+using HAProxy transaction-scoped variables.
+
+Variable names are normalized: dashes and spaces are replaced with special
+characters to comply with HAProxy variable naming rules. A meta-variable
+tracks the list of context variable names so they can be enumerated for
+extraction.
+
+Key functions:
+ flt_otel_var_register() Registers a variable with HAProxy.
+ flt_otel_var_set() Sets a variable value.
+ flt_otel_vars_get() Reads all context variables into a text_map for
+ extraction.
+ flt_otel_vars_unset() Removes all context variables.
+
+
+12 Group Action Integration
+----------------------------------------------------------------------
+
+The "otel-group" HAProxy action allows triggering trace scopes from
+tcp-request, tcp-response, http-request, http-response and
+http-after-response rules:
+
+ tcp-request otel-group <filter-id> <group-name>
+ tcp-response otel-group <filter-id> <group-name>
+ http-request otel-group <filter-id> <group-name>
+ http-response otel-group <filter-id> <group-name>
+ http-after-response otel-group <filter-id> <group-name>
+
+ group.c implements:
+ flt_otel_group_parse(): Parses the action arguments.
+ flt_otel_group_check(): Resolves group and scope references.
+ flt_otel_group_action(): At runtime, finds the OTel filter in the stream,
+ iterates all scopes in the group, and calls
+ flt_otel_scope_run() for each.
+
+
+13 Memory Management
+----------------------------------------------------------------------
+
+ pool.c provides wrappers around HAProxy memory pools and standard
+ allocation:
+
+ flt_otel_pool_alloc() Allocates from a pool (if non-NULL and the requested
+ size fits) or via calloc.
+ flt_otel_pool_free() Returns memory to the pool or frees it.
+ flt_otel_pool_strndup() Duplicates a string via pool allocation.
+ flt_otel_trash_alloc() Acquires a trash buffer chunk.
+ flt_otel_trash_free() Releases a trash buffer chunk.
+
+Four pool heads are registered for hot-path structures:
+ - otel_scope_span (scope.c)
+ - otel_scope_context (scope.c)
+ - otel_runtime_context (scope.c)
+ - otel_span_context (filter.c, used by the C wrapper via otelc_ext_init
+ callback)
+
+The wrapper library's memory allocations are redirected through
+flt_otel_mem_malloc() / flt_otel_mem_free() which use the otel_span_context
+pool. This ensures OTel objects benefit from HAProxy's pool allocator.
+
+
+14 CLI Interface
+----------------------------------------------------------------------
+
+ cli.c registers commands under "flt-otel" for runtime control:
+ - Setting the debug level.
+ - Enabling/disabling the filter on the fly.
+
+Logging can be independently controlled via the instrumentation's logging
+flags (ON, NOLOGNORM). Log output goes to the log servers configured in the
+instrumentation block.
+
+
+15 Debug Infrastructure
+----------------------------------------------------------------------
+
+When compiled with OTEL_DEBUG=1 (DEBUG_OTEL defined), the filter enables:
+
+ - Additional flt_ops callbacks: stream_set_backend, deinit_per_thread,
+ http_headers, http_payload, http_end, http_reset, http_reply, tcp_payload.
+ In non-debug builds these are set to NULL. (Note: stream_start and
+ stream_stop are always registered because they fire the on-stream-start
+ and on-stream-stop events.)
+
+ - The OTELC_DBG() macro produces debug output at various levels.
+
+ - flt_otel_scope_data_dump() dumps the complete scope data (baggage,
+ attributes, events, links, status) for inspection.
+
+ - Event usage counters (per-event htx_is_empty statistics) are maintained and
+ printed at deinit.
+
+ - Pool size information is printed at startup.
+
+The debug level is a bitmask that can be adjusted at runtime via the CLI.
+
+
+16 Test Infrastructure
+----------------------------------------------------------------------
+
+16.1 Test Scenarios
+
+ sa Standalone: comprehensive test exercising all request and response
+ events, span links (both inline and standalone syntax), events with data
+ capture, baggage, and the full span hierarchy from client session start
+ to server session end.
+
+ fe Frontend-only: tests the request-side span chain with context injection
+ into HTTP headers.
+
+ be Backend-only: tests context extraction from HTTP headers and
+ response-side processing. Designed to run as the backend of
+ the fe/ test.
+
+ ctx Context propagation: deep nesting test that verifies context propagation
+ via both HTTP headers and HAProxy variables.
+
+ cmp Comparison: simplified configuration made for comparison with other
+ tracing implementations.
+
+ empty Minimal: validates that an empty configuration (only the
+ instrumentation block, no scopes) does not crash.
+
+16.2 Test Runners
+
+All runners are POSIX shell scripts (/bin/sh). They accept an optional HAProxy
+binary path and log to test/_logs/.
+
+ run-sa.sh Runs a single HAProxy instance with sa/ config.
+ run-cmp.sh Runs a single HAProxy instance with cmp/ config.
+ test-speed.sh Runs performance benchmarks for one or all configurations.
+ run-ctx.sh Runs a single HAProxy instance with ctx/ config.
+ run-fe-be.sh Launches two HAProxy instances (frontend on port 10080, backend
+ on port 11080) forming a trace propagation chain. Handles
+ graceful shutdown via SIGUSR1.
+
+ copy-yml.sh Transforms a template YAML configuration by replacing
+ placeholders with test-specific values (service names, file
+ suffixes, etc.).
+
+16.3 Exporter Configuration
+
+Each test directory contains an otel.yml file configuring three exporter types:
+ - OTLP file exporter (writes traces to local files).
+ - OTLP gRPC exporter (sends to localhost:4317).
+ - OTLP HTTP exporter (sends to localhost:4318 in JSON format).
+
+
+17 Notable Design Decisions
+----------------------------------------------------------------------
+
+ - Span memoization: flt_otel_scope_span_init() and
+ flt_otel_scope_context_init() return existing entries if one with the
+ same name already exists. This allows multiple scopes to contribute data
+ (attributes, events) to the same logical span across different analyzer
+ events.
+
+ - Lazy span creation: the OTel span object is created on first use in
+ flt_otel_scope_run_span(), not at scope_span_init time. This separates
+ the span identity (name, parent reference) from the actual OTel resource.
+
+ - Soft/hard error modes: in soft mode, errors are logged but the stream
+ continues with tracing effectively abandoned for that span. In hard mode,
+ the filter disables itself for the rest of the stream. Either way, stream
+ processing is never interrupted by a tracing failure (FLT_OTEL_RET_OK is
+ always returned).
+
+ - Rate limiting uses a uint32 representation of a percentage
+ (FLT_OTEL_FLOAT_U32), compared against ha_random32() for uniform
+ distribution without floating-point at runtime.
+
+ - Server-unavailable fallback: if the backend was never reached (no response
+ analyzers executed), the on-server-unavailable event is fired at client
+ session end to ensure all spans are properly closed.
+
+ - Custom memory allocator: the C wrapper's allocations are routed through
+ HAProxy memory pools via otelc_ext_init(), keeping OTel objects in the
+ same allocation domain as the rest of the filter.
+
+ - Thread integration: flt_otel_thread_id() returns the HAProxy tid, ensuring
+ the wrapper's thread-local operations map to HAProxy worker threads.
+
+
+18 Tracer, Span and Metrics Internals
+----------------------------------------------------------------------
+
+This chapter describes the end-to-end lifecycle of the tracer and meter
+objects, the runtime span management model, and the metric instrument
+recording pipeline.
+
+18.1 Tracer Provider Initialization
+
+The tracer provider is set up during the proxy-level flt_otel_ops_init()
+callback, which delegates to flt_otel_lib_init() (filter.c).
+The initialization sequence is as follows:
+
+ 1. Version check: OTELC_IS_VALID_VERSION() verifies that the
+ OpenTelemetry C wrapper library version matches the header files.
+
+ 2. Configuration path: the relative path from the "config" keyword in
+ the instrumentation section is resolved to an absolute path using
+ getcwd() + snprintf().
+
+ 3. SDK initialization: otelc_init(path, err) loads the YAML
+ configuration file and sets up the SDK exporters, samplers,
+ processors and metric readers.
+
+ 4. Tracer creation: otelc_tracer_create(err) allocates the tracer
+ handle and stores it in instr->tracer.
+
+ 5. Meter creation: otelc_meter_create(err) allocates the meter handle
+ and stores it in instr->meter.
+
+ 6. Logger creation: otelc_logger_create(err) allocates the logger
+ handle and stores it in instr->logger.
+
+ 7. Extension callbacks: on success, otelc_ext_init() registers custom
+ memory allocation (flt_otel_mem_malloc / flt_otel_mem_free) and
+ thread-id (flt_otel_thread_id) callbacks so that OTel SDK objects
+ use HAProxy memory pools and thread numbering.
+
+ 8. Log handler: otelc_log_set_handler() installs a callback that
+ counts SDK diagnostic messages via the flt_otel_drop_cnt counter.
+
+All three handles are stored in the flt_otel_conf_instr structure
+(conf.h):
+
+ struct flt_otel_conf_instr {
+ ...
+ struct otelc_tracer *tracer; /* The OpenTelemetry tracer handle. */
+ struct otelc_meter *meter; /* The OpenTelemetry meter handle. */
+ struct otelc_logger *logger; /* The OpenTelemetry logger handle. */
+ ...
+ };
+
+18.2 Per-Thread Tracer, Meter and Logger Startup
+
+The flt_otel_ops_init_per_thread() callback (filter.c) starts the
+tracer, meter and logger background threads on the first call:
+
+ if (!(fconf->flags & FLT_CFG_FL_HTX)) {
+ retval = OTELC_OPS(conf->instr->tracer, start);
+ if (retval != OTELC_RET_ERROR) {
+ retval = OTELC_OPS(conf->instr->meter, start);
+ ...
+ }
+ if (retval != OTELC_RET_ERROR) {
+ retval = OTELC_OPS(conf->instr->logger, start);
+ ...
+ }
+ fconf->flags |= FLT_CFG_FL_HTX;
+ }
+
+The FLT_CFG_FL_HTX flag ensures that start is called only once, even
+when multiple proxies share the same filter configuration. If any
+start operation fails, the error string from the failing handle is
+forwarded via FLT_OTEL_ALERT.
+
+18.3 Tracer, Meter and Logger Shutdown
+
+At proxy deinit (flt_otel_ops_deinit, filter.c), the tracer, meter
+and logger are destroyed in a single call:
+
+ otelc_deinit(&((*conf)->instr->tracer), &((*conf)->instr->meter), &((*conf)->instr->logger));
+
+This flushes any pending spans, metric data and log records to the
+configured exporters, then releases the SDK resources. The full
+configuration tree is freed immediately after via flt_otel_conf_free().
+
+18.4 Span Lifecycle
+
+Spans progress through four phases: identity allocation, OTel span
+creation, data population, and completion.
+
+18.4.1 Span Identity Allocation
+
+When a scope containing a span definition executes for the first time,
+flt_otel_scope_span_init() (scope.c) allocates a scope_span
+entry from the otel_scope_span pool and inserts it into the runtime
+context's spans list:
+
+ retptr = flt_otel_pool_alloc(pool_head_otel_scope_span, ...);
+ retptr->id = id; /* Borrowed from config. */
+ retptr->id_len = id_len;
+ retptr->smp_opt_dir = dir;
+ retptr->ref_span = ref_span; /* Resolved parent span. */
+ retptr->ref_ctx = ref_ctx; /* Resolved parent context. */
+ LIST_INSERT(&(rt_ctx->spans), &(retptr->list));
+
+The parent reference (ref_id) is resolved at this point by searching the
+runtime context's spans list first, then the contexts list. If the
+parent name cannot be found in either list, an error is returned and the
+span is not created.
+
+Memoization: if a span with the same name already exists in
+rt_ctx->spans, the existing entry is returned without allocation. This
+allows multiple scopes (across different analyzer events) to contribute
+attributes, events and other data to the same logical span.
+
+18.4.2 OTel Span Creation (Lazy)
+
+The actual OTel span object is created lazily on first use in
+flt_otel_scope_run_span() (event.c):
+
+ if (span->span == NULL) {
+ span->span = OTELC_OPS(conf->instr->tracer,
+ start_span_with_options, span->id,
+ span->ref_span, span->ref_ctx,
+ ts_steady, ts_system, OTELC_SPAN_KIND_SERVER);
+ }
+
+The arguments are:
+
+ span->id The operation name (string identifier from config).
+ span->ref_span The parent span pointer (NULL if root or no parent).
+ span->ref_ctx The parent span context (from extracted context).
+ ts_steady Monotonic timestamp (CLOCK_MONOTONIC) for duration.
+ ts_system Wall-clock timestamp (CLOCK_REALTIME) for events.
+ OTELC_SPAN_KIND_SERVER Fixed span kind for all HAProxy spans.
+
+This separation between identity allocation and OTel creation means the
+span name, parent references and pool entry exist before the OTel
+resource is allocated. Subsequent scope executions that reference the
+same span name find the existing entry (via memoization) and add their
+data to the already-created OTel span.
+
+18.4.3 Span Data Population
+
+After creation, flt_otel_scope_run_span() (event.c) populates
+the span with data collected during scope execution:
+
+ Links (event.c):
+ Each resolved link is added via span->add_link(span, link_span,
+ link_context, NULL, 0). Links associate the span with other spans
+ or contexts without establishing a parent-child relationship. The
+ last two arguments (attributes array and count) are always NULL/0.
+
+ Baggage (event.c):
+ span->set_baggage_kv_n(data->baggage.attr, data->baggage.cnt)
+ sets key-value baggage items propagated across service boundaries.
+
+ Attributes (event.c):
+ span->set_attribute_kv_n(data->attributes.attr, data->attributes.cnt)
+ sets key-value span attributes evaluated from HAProxy sample
+ expressions.
+
+ Events (event.c):
+ For each event in data->events (iterated in reverse insertion order):
+ span->add_event_kv_n(event->name, ts_system, event->attr, event->cnt)
+ adds a named event with a wall-clock timestamp and key-value
+ attributes.
+
+ Status (event.c):
+ span->set_status(data->status.code, data->status.description)
+ sets the span's status code and description string. Only one status
+ per event is allowed.
+
+18.4.4 Span Context Injection
+
+After populating the span, if the configuration contains an "inject"
+directive (conf_span->ctx_id is non-NULL), the span context is
+serialized for downstream propagation (event.c).
+
+ flt_otel_inject_http_headers() serializes the span context into an
+ otelc_http_headers_writer, producing a text_map of key-value pairs.
+ For each pair, depending on the ctx_flags:
+
+ FLT_OTEL_CTX_USE_HEADERS:
+ flt_otel_http_header_set() writes the header into the HTX message.
+
+ FLT_OTEL_CTX_USE_VARS (requires OTEL_USE_VARS=1):
+ flt_otel_var_register() + flt_otel_var_set() store the value
+ in a HAProxy transaction variable.
+
+ Both storage types can be used simultaneously on the same span.
+
+18.4.5 Span Completion
+
+Spans are ended through the marking mechanism described in chapter 7.5.
+The actual end call in flt_otel_scope_finish_marked() (scope.c) is:
+
+ OTELC_OPSR(span->span, end_with_options,
+ ts_finish, OTELC_SPAN_STATUS_IGNORE, NULL);
+
+The arguments are the monotonic timestamp, a status hint (IGNORE means
+"do not override the status already set on the span"), and NULL for
+error string. After end_with_options returns, the OTELC_OPSR macro
+NULLs the span pointer, making the entry eligible for removal by
+flt_otel_scope_free_unused().
+
+On stream detach, flt_otel_runtime_context_free() (scope.c)
+force-ends any remaining active spans with the current monotonic
+timestamp and frees all pool entries.
+
+18.5 Metric Instruments
+
+The filter supports the full set of OpenTelemetry metric instrument
+types through a two-form configuration model: "create" form instruments
+define the instrument, and "update" form instruments record measurements
+against it.
+
+18.5.1 Instrument Types
+
+The following instrument types are available (parser.h):
+
+ cnt_int Counter (uint64)
+ hist_int Histogram (uint64)
+ udcnt_int UpDownCounter (int64)
+ gauge_int Gauge (int64)
+
+Observable (asynchronous) instruments are not supported. The OTel SDK invokes
+their callbacks from an external background thread that is not a HAProxy
+thread. HAProxy sample fetches rely on internal per-thread-group state and
+return incorrect results when called from a non-HAProxy thread.
+
+Double-precision types are not supported because HAProxy sample fetches do not
+return double values.
+
+ Special:
+ update Update-form instrument (records measurements)
+
+Each create-form instrument carries a description, unit, aggregation type,
+sample expression list, and optional histogram bucket boundaries. Each
+update-form instrument carries a reference to its create-form counterpart
+and an attribute key-value array for per-scope dimensions.
+
+18.5.2 Instrument Configuration Structure
+
+The flt_otel_conf_instrument structure (conf.h) holds:
+
+ idx Meter instrument index. Initially set to
+ OTELC_METRIC_INSTRUMENT_UNSET (-1). Transitions to
+ OTELC_METRIC_INSTRUMENT_PENDING (-2) during creation,
+ then to the positive meter index on success.
+ type The otelc_metric_instrument_t type constant, or
+ OTELC_METRIC_INSTRUMENT_UPDATE (0xff) for update-form.
+ aggr_type The otelc_metric_aggregation_type_t constant.
+ Initially OTELC_METRIC_AGGREGATION_UNSET (-1).
+ description Instrument description string (create-form only).
+ unit Instrument unit string (create-form only).
+ samples List of sample expressions for the instrument value.
+ bounds Histogram bucket boundaries array (create-form only).
+ bounds_num Number of histogram bucket boundaries.
+ attr Key-value attribute array (update-form only).
+ attr_len Number of attributes.
+ ref Pointer to the create-form instrument (update-form only).
+
+18.5.3 Meter Initialization and Startup
+
+The meter handle is created alongside the tracer in flt_otel_lib_init()
+(filter.c) via otelc_meter_create(err) and started per-thread
+in flt_otel_ops_init_per_thread() (filter.c) via
+OTELC_OPS(conf->instr->meter, start). The meter background thread
+handles periodic collection and export of metric data.
+
+18.5.4 Instrument Creation and Recording
+
+Metric instrument processing is performed by
+flt_otel_scope_run_instrument() (event.c), which runs in two
+passes during scope execution.
+
+ Pass 1 -- Create-form instruments (event.c):
+
+ Iterates all instruments in the scope. For each create-form
+ instrument whose idx is OTELC_METRIC_INSTRUMENT_UNSET:
+
+ a. Thread-safe one-time creation: HA_ATOMIC_CAS transitions the idx
+ from UNSET to PENDING. If the CAS fails (another thread is
+ already creating this instrument), the current thread skips it.
+
+ b. Instrument creation: meter->create_instrument() is called with
+ the instrument name, description, unit, type and callback data.
+ On success, the returned index is stored atomically; on failure,
+ the idx is reset to UNSET. If the instrument has an explicit
+ aggregation type or histogram bucket boundaries, meter->add_view()
+ is called before instrument creation to register a view with the
+ configured aggregation strategy and optional bounds. When bounds
+ are present without an explicit aggregation type, histogram
+ aggregation is used automatically for backward compatibility.
+
+ Pass 2 -- Update-form instruments (event.c):
+
+ Iterates all instruments again. For each update-form instrument:
+
+ a. Reference validation: the ref pointer must be non-NULL (resolved
+ at check time to the create-form instrument).
+
+ b. Index check: if the create-form instrument's idx is still
+ negative (UNUSED or PENDING), the measurement is skipped.
+
+ c. Recording: flt_otel_scope_run_instrument_record() evaluates the
+ sample expression, converts it to an otelc_value, and calls
+ meter->update_instrument_kv_n(idx, &value, attr, attr_len).
+
+18.5.5 Sample Evaluation for Metrics
+
+The recording function flt_otel_scope_run_instrument_record()
+(event.c) supports two evaluation paths:
+
+ Standard path: evaluates sample_process() on the first expression in
+ the create-form instrument's samples list, using the stream's backend,
+ session and direction context.
+
+ Log-format path: if sample->lf_used is set, allocates a temporary
+ buffer of global.tune.bufsize, calls build_logline() to evaluate the
+ log-format expression, and presents the result as an SMP_T_STR sample.
+
+Both paths converge on flt_otel_sample_to_value(), which converts the
+HAProxy sample data to an otelc_value. Metric instruments require
+numeric values (INT64); if the conversion produces
+OTELC_VALUE_DATA (string), a warning is logged and the measurement is
+rejected.
+
+
+18.5.6 Instrument Lifecycle Summary
+
+ Configuration time:
+ idx = OTELC_METRIC_INSTRUMENT_UNSET (-1)
+ type = instrument type constant
+
+ First scope execution (any thread):
+ idx transitions: UNSET -> PENDING -> meter_index (success)
+ UNSET -> PENDING -> UNSET (failure)
+
+ Subsequent scope executions:
+ Create-form: skipped (idx is already a valid meter index).
+ Update-form: evaluates samples and records via meter API.
+
+ Shutdown:
+ otelc_deinit() flushes and destroys tracer, meter and logger,
+ including all registered instruments and their callbacks.
+
+
+18.6 Log Records
+
+The filter supports OpenTelemetry log records via the "log-record"
+keyword inside otel-scope sections. Each log record is emitted through
+the OTel logger at a configured severity level, with an evaluated body,
+optional span correlation and optional key-value attributes.
+
+18.6.1 Log Record Configuration Structure
+
+The flt_otel_conf_log_record structure (conf.h) holds:
+
+ severity The otelc_log_severity_t severity level.
+ event_id Optional numeric event identifier (int64).
+ event_name Optional event name string.
+ span Optional span reference name (resolved at runtime).
+ attr Key-value attribute array (from "attr" keywords).
+ attr_len Number of attributes.
+ samples List of sample expressions for the body.
+
+The samples list contains exactly one flt_otel_conf_sample entry, which
+in turn holds either a list of bare sample expressions or a single
+log-format expression (when the value contains "%[").
+
+18.6.2 Log Record Emission
+
+Log record processing is performed by flt_otel_scope_run_log_record()
+(event.c), called from flt_otel_scope_run() after metric instrument
+processing and before span finishing.
+
+For each configured log record the function performs:
+
+ 1. Severity check: OTELC_OPS(logger, enabled, severity) tests whether
+ the logger accepts records at this severity. If not, the entry is
+ skipped. The threshold is controlled by the "min_severity" option
+ in the YAML logs signal configuration.
+
+ 2. Body evaluation: the single sample entry is evaluated using one of
+ two paths:
+
+ Log-format path (sample->lf_used is true):
+ A temporary buffer of global.tune.bufsize is allocated and
+ build_logline() evaluates the log-format expression into it.
+
+ Bare sample expression path:
+ Each expression in sample->exprs is evaluated via
+ sample_process() and converted to a string via
+ flt_otel_sample_to_str(). Results are concatenated into a
+ single buffer.
+
+ 3. Span resolution: if conf_log->span is non-NULL, the runtime
+ context's spans list is searched for a scope_span with a matching
+ name. If found, the OTel span pointer is captured for correlation.
+ A missing span is non-fatal -- a NOTICE warning is logged and the
+ record is emitted without span correlation.
+
+ 4. Emission: logger->log_span() is called with the severity, event_id,
+ event_name, resolved span (or NULL), wall-clock timestamp,
+ attributes and the evaluated body string.
+
+18.6.3 Logger Lifecycle Summary
+
+ Proxy init (flt_otel_lib_init):
+ otelc_logger_create() allocates the logger handle.
+
+ Per-thread init (flt_otel_ops_init_per_thread):
+ logger->start() launches the logger background thread.
+
+ Scope execution (flt_otel_scope_run):
+ flt_otel_scope_run_log_record() emits records via logger->log_span().
+
+ Shutdown (flt_otel_ops_deinit):
+ otelc_deinit() flushes pending log records and destroys the logger.
--- /dev/null
+OpenTelemetry filter -- miscellaneous notes
+==============================================================================
+
+1 Parsing sample expressions in HAProxy
+------------------------------------------------------------------------------
+
+HAProxy provides two entry points for turning a configuration string into an
+evaluable sample expression.
+
+
+1.1 sample_parse_expr()
+..............................................................................
+
+Parses a bare sample-fetch name with an optional converter chain. The input is
+the raw expression without any surrounding syntax.
+
+ Declared in: include/haproxy/sample.h
+ Defined in: src/sample.c
+
+ struct sample_expr *sample_parse_expr(char **str, int *idx, const char *file, int line, char **err_msg, struct arg_list *al, char **endptr);
+
+The function reads from str[*idx] and advances *idx past the consumed tokens.
+
+Configuration example (otel-scope instrument keyword):
+
+ instrument my_counter "name" desc req.hdr(host),lower ...
+
+Here "req.hdr(host),lower" is a single configuration token that
+sample_parse_expr() receives directly. It recognises the fetch "req.hdr(host)"
+and the converter "lower" separated by a comma.
+
+
+1.2 parse_logformat_string()
+..............................................................................
+
+Parses a log-format string that may contain literal text mixed with sample
+expressions wrapped in %[...] delimiters.
+
+ Declared in: include/haproxy/log.h
+ Defined in: src/log.c
+
+ int parse_logformat_string(const char *fmt, struct proxy *curproxy, struct lf_expr *lf_expr, int options, int cap, char **err);
+
+Configuration example (HAProxy log-format directive):
+
+ log-format "host=%[req.hdr(host),lower] status=%[status]"
+
+The %[...] wrapper tells parse_logformat_string() where each embedded sample
+expression begins and ends. The text outside the brackets ("host=", " status=")
+is emitted as-is.
+
+
+1.3 Which one to use
+..............................................................................
+
+Use sample_parse_expr() when the configuration token is a single, standalone
+sample expression (no surrounding text). This is the case for the otel filter
+keywords such as "attribute", "event", "baggage", "status", "value", and
+similar.
+
+Use parse_logformat_string() when the value is a free-form string that may mix
+literal text with zero or more embedded expressions.
+
+
+2 Signal keywords
+------------------------------------------------------------------------------
+
+The OTel filter configuration uses one keyword per signal to create or update
+signal-specific objects. The keyword names follow the OpenTelemetry
+specification's own terminology rather than using informal synonyms.
+
+ Signal Keyword Creates / updates
+ -------- ----------- ------------------------------------------
+ Tracing span A trace span.
+ Metrics instrument A metric instrument (counter, gauge, ...).
+ Logging log-record A log record.
+
+The tracing keyword follows the same logic. A "trace" is the complete
+end-to-end path of a request through a distributed system, composed of one or
+more "spans". Each span represents a single unit of work within that trace.
+The configuration operates at the span level: it creates individual spans, sets
+their parent-child relationships, and attaches attributes and events. Using
+"trace" as the keyword would be imprecise because one does not configure a trace
+directly; one configures the spans that collectively form a trace.
+
+The metrics keyword is analogous. In the OpenTelemetry data model the
+terminology is layered: a "metric" is the aggregated output that the SDK
+produces after processing recorded measurements, while an "instrument" is the
+concrete object through which those measurements are recorded -- a counter,
+histogram, gauge, or up-down counter. The configuration operates at the
+instrument level: it creates an instrument of a specific type and records values
+through it. Using "metric" as the keyword would be imprecise because one does
+not configure a metric directly; one configures an instrument that yields
+metrics.
+
+The logging keyword follows the same pattern. A "log" is the broad signal
+category, while a "log record" is a single discrete entry within that signal.
+The configuration operates at the log-record level: it creates individual log
+records with a severity, a body, and optional attributes and span context.
+Using "log" as the keyword would be imprecise because one does not configure a
+log stream directly; one configures the individual log records that comprise it.
projects / thirdparty / haproxy.git / commitdiff
