MEDIUM: threads: change the default max-threads-per-group value to 16
A lot of our subsystems start to be shared by thread groups now
(listeners, queues, stick-tables, stats, idle connections, LB algos).
This has allowed to recover the performance that used to be out of
reach on losely shared platforms (typically AMD EPYC systems), but in
parallel other large unified systems (Xeon and large Arm in general)
still suffer from the remaining contention when placing too many
threads in a group.
A first test running on a 64-core Neoverse-N1 processor with a single
backend with one server and no LB algo specifiied shows 1.58 Mrps with
64 threads per group, and 1.71 Mrps with 16 threads per group. The
difference is essentially spent updating stats counters everywhere.
Another test is the connection:close mode, delivering 85 kcps with
64 threads per group, and 172 kcps (202%) with 16 threads per group.
In this case it's mostly the more numerous listeners which improve
the situation as the change is mostly in the kernel:
So let's change the default value to 16. It also happens to match what's
used by default on EPYC systems these days.
This change was marked MEDIUM as it will increase the number of listening
sockets on some systems, to match their counter parts from other vendors,
which is easier for capacity planning.
DOC: config: fix spelling of "max-threads-per-group" in the index
It was spelled "max-thread-per-group" (without 's'). No backport is
needed unless commit 7e22d9c484 ("MEDIUM: cpu-topo: Add a new
"max-threads-per-group" global keyword") and its possible successors
are backported.
Released version 3.4-dev9 with the following main changes :
- DOC: config: fix ambiguous info in log-steps directive description
- MINOR: filters: add filter name to flt_conf struct
- MEDIUM: filters: add "filter-sequence" directive
- REGTESTS: add a test for "filter-sequence" directive
- Revert "CLEANUP: tcpcheck: Don't needlessly expose proxy_parse_tcpcheck()"
- MINOR: tcpcheck: reintroduce proxy_parse_tcpcheck() symbol
- BUG/MEDIUM: haterm: Move all init functions of haterm in haterm_init.c
- BUG/MEDIUM: mux-h1: Disable 0-copy forwarding when draining the request
- MINOR: servers: The right parameter for idle-pool.shared is "full"
- DOC: config: Fix two typos in the server param "healthcheck" description
- BUG/MINOR: http-act: fix a typo in the "pause" action error message
- MINOR: tcpcheck: Reject unknown keyword during parsing of healthcheck section
- BUG/MEDIUM: tcpcheck/server: Fix parsing of healthcheck param for dynamic servers
- BUG/MINOR: counters: fix unexpected 127 char GUID truncation for shm-stats-file objects
- BUG/MEDIUM: tcpcheck: Properly retrieve tcpcheck type to install the best mux
- BUG/MEDIUM: payload: validate SNI name_len in req.ssl_sni
- BUG/MEDIUM: jwe: fix NULL deref crash with empty CEK and non-dir alg
- BUG/MEDIUM: jwt: fix heap overflow in ECDSA signature DER conversion
- BUG/MEDIUM: jwe: fix memory leak in jwt_decrypt_secret with var argument
- BUG: hlua: fix stack overflow in httpclient headers conversion
- BUG/MINOR: hlua: fix stack overflow in httpclient headers conversion
- BUG/MINOR: hlua: fix format-string vulnerability in Patref error path
- BUG/MEDIUM: chunk: fix typo allocating small trash with bufsize_large
- BUG/MEDIUM: chunk: fix infinite loop in get_larger_trash_chunk()
- BUG/MINOR: peers: fix OOB heap write in dictionary cache update
- CI: VTest build with git clone + cache
- BUG/MEDIUM: connection: Wake the stconn on error when failing to create mux
- CI: github: update to cache@v5
- Revert "BUG: hlua: fix stack overflow in httpclient headers conversion"
- CI: github: fix vtest path to allow correct caching
- CI: github: add the architecture to the cache key for vtest2
- MEDIUM: connections: Really enforce mux protocol requirements
- MINOR: tools: Implement net_addr_type_is_quic()
- MEDIUM: check: Revamp the way the protocol and xprt are determined
- BUG/MAJOR: slz: always make sure to limit fixed output to less than worst case literals
- MINOR: lua: add tune.lua.openlibs to restrict loaded Lua standard libraries
- REGTESTS: lua: add tune.lua.openlibs to all Lua reg-tests
- BUG/MINOR: resolvers: fix memory leak on AAAA additional records
- BUG/MINOR: spoe: fix pointer arithmetic overflow in spoe_decode_buffer()
- BUG/MINOR: http-act: validate decoded lengths in *-headers-bin
- BUG/MINOR: haterm: Return the good start-line for 100-continue interim message
- BUG/MEDIUM: samples: Fix handling of SMP_T_METH samples
- BUG/MINOR: sample: fix info leak in regsub when exp_replace fails
- BUG/MEDIUM: mux-fcgi: prevent record-length truncation with large bufsize
- BUG/MINOR: hlua: fix use-after-free of HTTP reason string
- BUG/MINOR: mux-quic: fix potential NULL deref on qcc_release()
- BUG/MINOR: quic: increment pos pointer on QMux transport params parsing
- MINOR: xprt_qstrm: implement Rx buffering
- MINOR: xprt_qstrm/mux-quic: handle extra QMux frames after params
- MINOR: xprt_qstrm: implement Tx buffering
- MINOR: xprt_qstrm: handle connection errors
- MEDIUM: mux-quic: implement QMux record parsing
- MEDIUM: xprt_qstrm: implement QMux record parsing
- MEDIUM: mux-quic/xprt_qstrm: implement QMux record emission
- DOC: update draft link for QMux protocol
- BUG/MINOR: do not crash on QMux reception of BLOCKED frames
- Revert "BUG/MEDIUM: haterm: Move all init functions of haterm in haterm_init.c"
- BUG/MEDIUM: haterm: Properly initialize the splicing support for haterm
- BUG/MINOR: mux_quic: prevent QMux crash on qcc_io_send() error path
- BUG/MINOR: xprt_qstrm: do not parse record length on read again
- MEDIUM: otel: added OpenTelemetry filter skeleton
- MEDIUM: otel: added configuration and utility layer
- MEDIUM: otel: added configuration parser and event model
- MEDIUM: otel: added post-parse configuration check
- MEDIUM: otel: added memory pool and runtime scope layer
- MEDIUM: otel: implemented filter callbacks and event dispatcher
- MEDIUM: otel: wired OTel C wrapper library integration
- MEDIUM: otel: implemented scope execution and span management
- MEDIUM: otel: added context propagation via carrier interfaces
- MEDIUM: otel: added HTTP header operations for context propagation
- MEDIUM: otel: added HAProxy variable storage for context propagation
- MINOR: otel: added prefix-based variable scanning
- MEDIUM: otel: added CLI commands for runtime filter management
- MEDIUM: otel: added group action for rule-based scope execution
- MINOR: otel: added log-format support to the sample parser and runtime
- MINOR: otel: test: added test and benchmark suite for the OTel filter
- MINOR: otel: added span link support
- MINOR: otel: added metrics instrument support
- MINOR: otel: added log-record signal support
- MINOR: otel: test: added full-event test config
- DOC: otel: added documentation
- DOC: otel: test: added test README-* files
- DOC: otel: test: added speed test guide and benchmark results
- DOC: otel: added cross-cutting design patterns document
- MINOR: otel: added flt_otel_sample_eval and exposed flt_otel_sample_add_kv
- MINOR: otel: changed log-record attr to use sample expressions
- MINOR: otel: changed instrument attr to use sample expressions
- DOC: otel: added README.md overview document
- CLEANUP: ot: use the item API for the variables trees
- BUG/MINOR: ot: removed dead code in flt_ot_parse_cfg_str()
- BUG/MINOR: ot: fixed wrong NULL check in flt_ot_parse_cfg_group()
- BUILD: ot: removed explicit include path when building opentracing filter
- MINOR: ot: renamed the variable dbg_indent_level to flt_ot_dbg_indent_level
- CI: Drop obsolete `packages: write` permission from `quic-interop-*.yml`
- CI: Consistently add a top-level `permissions` definition to GHA workflows
- CI: Wrap all `if:` conditions in `${{ }}`
- CI: Fix regular expression escaping in matrix.py
- CI: Update to actions/checkout@v6
- CI: Simplify version extraction with `haproxy -vq`
- CI: Merge `aws-lc.yml` and `aws-lc-fips.yml` into `aws-lc.yml`
- CI: Merge `aws-lc-template.yml` into `aws-lc.yml`
- CI: Consistently set up VTest with `./.github/actions/setup-vtest`
- MINOR: mux_quic: remove duplicate QMux local transport params
- CI: github: add bash to the musl job
- BUG/MINOR: quic: do not use hardcoded values in QMux TP frame builder
- BUG/MINOR: log: Fix error message when using unavailable fetch in logfmt
- CLEANUP: log: Return `size_t` from `sess_build_logline_orig()`
- CLEANUP: stream: Explain the two-step initialization in `stream_generate_unique_id()`
- CLEANUP: stream: Reduce duplication in `stream_generate_unique_id()`
- CLEANUP: http_fetch: Use local `unique_id` variable in `smp_fetch_uniqueid()`
- CI: build WolfSSL job with asan enabled
- MINOR: tools: memvprintf(): remove <out> check that always true
- BUG/MEDIUM: cli: Properly handle too big payload on a command line
- REGTESTS: Never reuse server connection in reg-tests/jwt/jwt_decrypt.vtc
- MINOR: errors: remove excessive errmsg checks
- BUG/MINOR: haterm: preserve the pipe size margin for splicing
- MEDIUM: acme: implement dns-persist-01 challenge
- MINOR: acme: extend resolver-based DNS pre-check to dns-persist-01
- DOC: configuration: document dns-persist-01 challenge type and options
- BUG/MINOR: acme: read the wildcard flag from the authorization response
- BUG/MINOR: acme: don't pass NULL into format string
- BUG/MINOR: haterm: don't apply the default pipe size margin twice
- CLEANUP: Make `lf_expr` parameter of `sess_build_logline_orig()` const
- MINOR: Add `generate_unique_id()` helper
- MINOR: Allow inlining of `stream_generate_unique_id()`
- CLEANUP: log: Stop touching `struct stream` internals for `%ID`
- MINOR: check: Support generating a `unique_id` for checks
- MINOR: http_fetch: Add support for checks to `unique-id` fetch
- MINOR: acme: display the type of challenge in ACME_INITIAL_DELAY
- MINOR: mjson: reintroduce mjson_next()
- CI: Remove obsolete steps from musl.yml
- CI: Use `sh` in `actions/setup-vtest/action.yml`
- CI: Sync musl.yml with vtest.yml
- CI: Integrate Musl build into vtest.yml
- CI: Use `case()` function
- CI: Generate vtest.yml matrix on `ubuntu-slim`
- CI: Run contrib.yml on `ubuntu-slim`
- CI: Use `matrix:` in contrib.yml
- CI: Build `dev/haring/` as part of contrib.yml
- MINOR: htx: Add helper function to get type and size from the block info field
- BUG/MEDIUM: htx: Properly handle block modification during defragmentation
- BUG/MEDIUM: htx: Don't count delta twice when block value is replaced
- MINOR: ssl: add TLS 1.2 values in HAPROXY_KEYLOG_XX_LOG_FMT
- EXAMPLES: ssl: keylog entries are greater than 1024
- BUILD: Makefile: don't forget to also delete haterm on make clean
- MINOR: stats: report the number of thread groups in "show info"
- CLEANUP: sample: fix the comment regarding the range of the thread sample fetch
- MINOR: sample: return the number of the current thread group
- MINOR: sample: add new sample fetch functions reporting current CPU usage
- BUG/MEDIUM: peers: trash of expired entries delayed after fullresync
- DOC: remove the alpine/musl status job image
- MINOR: mux-quic: improve documentation for qcs_attach_sc()
- MINOR: mux-quic: reorganize code for app init/shutdown
- MINOR: mux-quic: perform app init in case of early shutdown
- MEDIUM: quic: implement fe.stream.max-total
- MINOR: mux-quic: close connection when reaching max-total streams
- REGTESTS: add QUIC test for max-total streams setting
- MEDIUM: threads: start threads by groups
- MINOR: acme: opportunistic DNS check for dns-persist-01 to skip challenge-ready steps
- BUG/MINOR: acme: fix fallback state after failed initial DNS check
- CLEANUP: acme: no need to reset ctx state and http_state before nextreq
- BUG/MINOR: threads: properly set the number of tgroups when non using policy
BUG/MINOR: threads: properly set the number of tgroups when non using policy
When nbthread is set, the CPU policies are not used and do not set
nbthread nor nbtgroups. When back into thread_detect_count(), these
are set respectively to thr_max and 1. The problem which becomes very
visible with max-threads-per-group, is that setting this one in
combination with nbthreads results in only one group with the calculated
number of threads per group. And there's not even a warning. So basically
a configuration having:
global
nbthread 64
max-threads-per-group 8
would only start 8 threads.
In this case, grp_min remains valid and should be used, so let's just
change the assignment so that the number of groups is always correct.
A few ifdefs had to move because the calculations were only made for
the USE_CPU_AFFINITY case. Now these parts have been refined so that
all the logic continues to apply even without USE_CPU_AFFINITY.
One visible side effect is that setting nbthread above 64 will
automatically create the associated number of groups even when
USE_CPU_AFFINITY is not set. Previously it was silently changed
to match the per-group limit.
Ideally this should be backported to 3.2 where the issue was
introduced, though it may change the behavior of configs that were
silently being ignored (e.g. "nbthread 128"), so the backport should
be considered with care. At least 3.3 should have it because it uses
cpu-policy by default so it's only for failing cases that it would be
involved.
CLEANUP: acme: no need to reset ctx state and http_state before nextreq
The nextreq label already implement setting http_state to ACME_HTTP_REQ
and setting ctx->state to st. It is only needed to set the st variable
before jumping to nextreq.
BUG/MINOR: acme: fix fallback state after failed initial DNS check
When the opportunistic initial DNS check (ACME_INITIAL_RSLV_READY) fails,
the state machine was incorrectly transitioning to ACME_RSLV_RETRY_DELAY
instead of ACME_CLI_WAIT. This caused the challenge to enter the DNS retry
loop rather than falling back to the normal cond_ready flow that waits for
the CLI signal.
Also reorder ACME_CLI_WAIT in the state enum and trace switch to reflect
the actual execution order introduced in the previous commit: it comes after
ACME_INITIAL_RSLV_READY, not before ACME_INITIAL_RSLV_TRIGGER.
MINOR: acme: opportunistic DNS check for dns-persist-01 to skip challenge-ready steps
For dns-persist-01, the "_validation-persist.<domain>" TXT record is set once
and never changes between renewals. Add an initial opportunistic DNS check
(ACME_INITIAL_RSLV_TRIGGER / ACME_INITIAL_RSLV_READY states) that runs before
the challenge-ready conditions are evaluated. If all domains already have the
TXT record, the challenge is submitted immediately without going through the
cli/delay/dns challenge-ready steps, making renewals faster once the record is
in place.
The new ACME_RDY_INITIAL_DNS flag is automatically set for
dns-persist-01 in cond_ready.
Till now, threads were all started one at a time from thread 1. This
will soon cause us limitations once we want to reduce shared stuff
between thread groups.
Let's slightly change the startup sequence so that the first thread
starts one initial thread for each group, and that each of these
threads then starts all other threads from their group before switching
to the final task. Since it requires an intermediary step, we need to
store that threads' start function to access it from the group, so it
was put into the tgroup_info which still has plenty of room available.
It could also theoretically speed up the boot sequence, though in
practice it doesn't change anything because each thead's initialization
is made one at a time to avoid races during the early boot. However
ther is now a function in charge of starting all extra threads of a
group, and whih is called from this group.
REGTESTS: add QUIC test for max-total streams setting
Add a new QUIC regtest to test the new frontend stream.max-total
setting.
This test relies on two haproxy instances, as QUIC client and server.
New setting stream.max-total is set to 3 on the server side. In total, 6
requests are performed, with a check to ensure that a new connection has
been reopened for the last ones.
MINOR: mux-quic: close connection when reaching max-total streams
This commit completes the previous one which implements a new setting to
limit the number of streams usable by a client on a QUIC connection.
When the connection becomes idle after reaching this limit, it is
immediately closed. This is implemented by extending checks in
qcc_is_dead(). This results in a CONNECTION_CLOSE emission, which is
useful to free resources as soon as possible.
Implement a new setting to limit the total number of bidirectional
streams that the client may use on a single connection. By default, it
is set to 0 which means it is not limited at all.
If a positive value is configured, the client can only open a fixed
number of request streams per QUIC connection. Internally, this is
implemented in two steps :
* First, MAX_STREAMS_BIDI flow control advertizing will be reduced when
approaching the limit before being completely turned off when reaching
it. This guarantees that the client cannot exceed the limit without
violating the flow control.
* Second, when attaching the latest stream with ID matching max-total
setting, connection graceful shutdown is initiated. In HTTP/3, this
results in a GOAWAY emission. This allows the remaining streams to be
completed before the connection becomes completely idle.
MINOR: mux-quic: perform app init in case of early shutdown
Adds a qcc_app_init() call in qcc_app_shutdown(). This is necessary if
shutdown is performed early, before any invokation of qcc_io_send().
Currently, this should never occur in practice. However, this will
become necessary with the new settings tune.quic.fe.stream.max-total.
Indeed, when using a very small value, app-ops layer may be closed early
in the connection lifetime.
MINOR: mux-quic: reorganize code for app init/shutdown
Refactor code related to app-layer init/shutdown operations. In short,
qcc_shutdown() is renamed to qcc_app_shutdown(). It is also moved next
to qcc_app_init() to better reflect their link.
MINOR: mux-quic: improve documentation for qcs_attach_sc()
Complete function doc for qcs_attach_sc() by using the proper
terminology related to stream/stconn/sedesc. The purpose of this
function should be clearer now.
BUG/MEDIUM: peers: trash of expired entries delayed after fullresync
stksess_new has set the entry expire to the table expire delay,
if it is a new entry, set_entry inserts at that position in the expire
tree. There was a touch_remote updating the expire setting but the
tree's re-ordering is not designed to set back in the past resulting
to an entry that will be trashed only after a full table's expire delay
regardless the expire set on the stktsess.
This patch sets the newts expire before the call of 'set_entry'.
This way a new inserted entry is set directly at the right position
in the tree to trash the entry in time.
This patch should be backported on all supported branches and at
least v2.8
MINOR: sample: add new sample fetch functions reporting current CPU usage
Some features can automatically turn on or off depending on CPU usage,
but it's not easy to measure it. Let's provide 3 new sample fetch functions
reporting the CPU usage as measured inside haproxy during the previous
polling loop, and reported in "idle" stats header / "show info", or used
by tune.glitches.kill.cpu-usage, or maxcompcpuusage:
- cpu_usage_thr: CPU usage between 0 and 100 of the current thread, used
by functions above
- cpu_usage_grp: CPU usage between 0 and 100, averaged over all threads of
the same group as the current one.
- cpu_usage_proc: CPU usage between 0 and 100, averaged over all threads
of the current process
Note that the value will fluctuate since it only covers a few tens to
hundreds of requests of the last polling loop, but it reports what is
being used to take decisions.
It could also be used to disable some non-essential debugging/processing
under too high loads for example.
MINOR: sample: return the number of the current thread group
Just like we have a sample fetch function that returns the number of the
current thread, let's have the same with the thread group number. This
can be useful for troubleshooting, given that certain things are currently
per thread-group (e.g. idle backend connections, certain LB algos etc).
CLEANUP: sample: fix the comment regarding the range of the thread sample fetch
The comment says "between 1 and nbthread" while it's in fact between 0 and
nbthread-1 and this is also documented like this in the config manual. No
backport needed though it cannot hurt.
MINOR: stats: report the number of thread groups in "show info"
Since thread groups were enabled by default in 3.3, it has become an
important element of diagnostic that we're missing in "show info". Let's
add it under "NbThreadGroups".
BUILD: Makefile: don't forget to also delete haterm on make clean
haterm depends on the same source files as haproxy, yet it wasn't deleted
on "make clean", resulting in confusion when rebuilding and believing to
run the freshly built one. Let's just add it to the "clean" target. No
backport is needed since haterm is 3.4-only.
EXAMPLES: ssl: keylog entries are greater than 1024
Adjust the log size to 2048, the default 1024 bytes of a log line are
too small since f28dd15 ("MINOR: ssl: add TLS 1.2 values in
HAPROXY_KEYLOG_XX_LOG_FMT")
MINOR: ssl: add TLS 1.2 values in HAPROXY_KEYLOG_XX_LOG_FMT
Add the CLIENT_RANDOM line for TLS1.2 in HAPROXY_KEYLOG_FC_LOG_FMT and
HAPROXY_KEY_LOG_BC_FMT. These are useful to produce a keylog file
compatible with both TLS1.3 and TLS1.2.
BUG/MEDIUM: htx: Don't count delta twice when block value is replaced
A regression was introduced by the commit a8887e55a ("BUG/MEDIUM: htx: Fix
function used to change part of a block value when defrag").
When a block value was replaced and a defragmentation was performed, the
delta between the old value and the new one was counted twice. htx_defrag()
already is responsible to set the new size for the HTX message. So it must
not be performed in htx_replace_blk_value().
This patch must be backported with the commit above. So theorically to all
stable versions.
BUG/MEDIUM: htx: Properly handle block modification during defragmentation
A regression was introcuded by the commit 0c6f2207f ("MEDIUM: htx: Refactor
htx defragmentation to merge data blocks").
When a defragmentation is performed, it is possible to alter a block
size. The main usage is to prepare a block value replacement. However, since
the commit above, the change is no longer handled. The block info are
changed but the size of the message is not modified accordingly.
This patch depends on the commit "MINOR: htx: Add helper function to get
type and size from the block info field"
MINOR: htx: Add helper function to get type and size from the block info field
__htx_blkinfo_type() and __htx_blkinfo_size() function was added to return,
respectively, the type and the size from the block info field. The main
usage for these functions is internal to the htx code.
Tim Duesterhus [Mon, 13 Apr 2026 19:23:45 +0000 (21:23 +0200)]
CI: Use `matrix:` in contrib.yml
This makes it much easier to add additional "smoke-tests" to contrib.yml. The
previous set-up also didn't allow to easily see all failures when a single
build fails, because it would abort after any failed step.
Tim Duesterhus [Mon, 13 Apr 2026 19:23:42 +0000 (21:23 +0200)]
CI: Use `case()` function
GitHub Actions introduced a new `case()` function in January that works just
like HAProxy’s `iif()` converter (just with an arbitrary number of
expressions). It is more robust than chaining strings with `&&` and `||`,
because it includes proper type-checking.
see https://github.blog/changelog/2026-01-29-github-actions-smarter-editing-clearer-debugging-and-a-new-case-function/#write-more-expressive-expressions-with-a-case-function
Tim Duesterhus [Mon, 13 Apr 2026 19:23:41 +0000 (21:23 +0200)]
CI: Integrate Musl build into vtest.yml
With the previous sync, these two workflows perform almost the same steps and
both logically belong to "Run VTest tests". Integrate musl.yml into vtest.yml,
which will hopefully encourage future changes to consistently apply to all jobs
in that workflow.
Tim Duesterhus [Mon, 13 Apr 2026 17:37:31 +0000 (19:37 +0200)]
MINOR: http_fetch: Add support for checks to `unique-id` fetch
This allows to use the `unique-id` fetch within `tcp-check` or `http-check`
ruleset. The format is taken from the checked server's backend (which is
naturally inherited from the corresponding `defaults` section).
This is particularly useful with
http-check send ... hdr request-id %[unique-id]
to ensure all requests sent by HAProxy have a unique ID header attached.
Tim Duesterhus [Mon, 13 Apr 2026 17:37:29 +0000 (19:37 +0200)]
CLEANUP: log: Stop touching `struct stream` internals for `%ID`
Use the return value of `stream_generate_unique_id()` instead of relying on the
`unique_id` field of `struct stream` when handling the `%ID` log placeholder.
This also allowed to unify the "stream available" and "stream not available"
paths.
Tim Duesterhus [Mon, 13 Apr 2026 17:37:28 +0000 (19:37 +0200)]
MINOR: Allow inlining of `stream_generate_unique_id()`
With the introduction of the `generate_unique_id()` helper, the actual
complicated logic is sitting in a different file. Allow inlining of
`stream_generate_unique_id()`, so that callers can benefit from an abstraction
without hiding away the access of `strm->unique_id` behind a function call.
Tim Duesterhus [Mon, 13 Apr 2026 17:37:27 +0000 (19:37 +0200)]
MINOR: Add `generate_unique_id()` helper
This new function will handle the actual generation of the unique ID according
to a format. The caller is responsible to check that no unique ID is stored
yet.
BUG/MINOR: haterm: don't apply the default pipe size margin twice
Commit 6d16b11022 ("BUG/MINOR: haterm: preserve the pipe size margin
for splicing") solved the issue of pipe size being sufficient for the
vmsplice() call, but as Christopher pointed out, the ratio was applied
to the default size of 64k, so now it's applied twice, giving 100k
instead of 80k. Let's drop it from there.
BUG/MINOR: acme: read the wildcard flag from the authorization response
The wildcard field was declared and used when building the dns-persist-01
TXT record value (policy=wildcard suffix), but was never populated from
the server's authorization response. Add the missing mjson_get_bool() call
to read $.wildcard before saving auth->dns.
DOC: configuration: document dns-persist-01 challenge type and options
Document the dns-persist-01 challenge type under the challenge keyword,
the challenge-ready dns option (existence-only TXT check for dns-persist-01),
and the default challenge-ready value when challenge is dns-persist-01.
MINOR: acme: extend resolver-based DNS pre-check to dns-persist-01
Add challenge_type parameter to acme_rslv_start() to select the correct
DNS lookup prefix: _validation-persist.<domain> for dns-persist-01 and
_acme-challenge.<domain> for dns-01.
Default cond_ready to ACME_RDY_DNS|ACME_RDY_DELAY for dns-persist-01.
Extend ACME_CLI_WAIT to cover dns-persist-01 alongside dns-01.
In ACME_RSLV_READY, check only TXT record existence for dns-persist-01
since the resolver cannot parse multiple strings within a single TXT entry.
Mia Kanashi [Wed, 25 Feb 2026 22:07:55 +0000 (00:07 +0200)]
MEDIUM: acme: implement dns-persist-01 challenge
Implements draft DNS-PERSIST-01 challenge based on
https://datatracker.ietf.org/doc/html/draft-ietf-acme-dns-persist
Blog post: https://letsencrypt.org/2026/02/18/dns-persist-01
This challenge is designed to use preprovisioned DNS records,
unlike DNS-01 challenge it doesn't need per provider API integration.
In short instead of validating order by crafting a custom response
based on input recieved from ACME server, like other challenges do
in particular DNS-01, HTTP-01, TLS-ALPN-01, in this challenge you
authorize domain statically, ACME account key functions similar to
a private key and accounturi in the record functions like a public key,
ACME server verifies that account uri matches account key and authorizes
based on that. You only need to write DNS record one time,
accounturi binds to an account key, and will only change if new account
key is created, although it is possible to rotate account key without
changing account uri.
Main benefits of this challenge in contrast to DNS-01:
1. Security, no need to give reverse proxy write access to the DNS.
2. Simplicity, no complex per provider integrations like Lego needed.
3. Robustness, no worrying about DNS record cache each renewal.
It would be used like this:
1. generate an account key ahead of time
2. add required DNS record manually or automatically using IaC tools
3. start HAProxy with the same account key used
Intended way to use this challenge is with a code that will print
and maybe sets DNS records ahead of time. For example that could
be integrated into the IaC provisioning step. This challenge type
is extremely recent though, so those integrations are yet to be written.
It is possible to do this challenge without extra tools too,
with pebble / challtestsrv steps would be as following:
After starting HAProxy it will print required records in the logs.
With challtestsrv you can then set those records like this:
BUG/MINOR: haterm: preserve the pipe size margin for splicing
Originally in httpterm we used to allocate 5/4 of the size of a pipe to
permit to use vmsplice because there's some fragmentation or overhead
internally that requires to use a bit of margin. While this was initially
applied to haterm as well, it was accidentally lost with commit fb82dece47
("BUG/MEDIUM: haterm: Properly initialize the splicing support for haterm"),
resulting in errors about vmsplice() whenever tune.pipesize is set. Let's
enforce the ratio again.
I noticed some strange checks for presence of errmsg. Called functions
generate non-empty error message in case of failure, so a non-NULL address
of the error message is enough.
BUG/MEDIUM: cli: Properly handle too big payload on a command line
When command line is parsed, when the payload was too big the error was not
properly handled. Instead of leaving the parsing function to print the
error, we looped infinitly trying to parse remaining data.
When the command line is too big, we must exit the parsing function in
CLI_ST_PRINT_ERR state. Instead of exiting the function, we only left the
while loop, setting this way the cli applet in CLI_ST_PROMPT state.
Tim Duesterhus [Fri, 3 Apr 2026 21:29:01 +0000 (23:29 +0200)]
CLEANUP: http_fetch: Use local `unique_id` variable in `smp_fetch_uniqueid()`
Instead of relying on the implementation detail that
`stream_generate_unique_id()` will store the unique ID in `strm->unique_id` we
should use the returned value, especially since that one is already checked in
the `isttest()`.
Tim Duesterhus [Fri, 3 Apr 2026 21:28:58 +0000 (23:28 +0200)]
CLEANUP: stream: Explain the two-step initialization in `stream_generate_unique_id()`
This two-step initialization of `strm->unique_id` looks like a refactoring
target. Add a comment to prevent regressions of the fix in fb7b5c8a53cb4f19a223abd20660d47162aa8708.
Tim Duesterhus [Fri, 3 Apr 2026 21:28:57 +0000 (23:28 +0200)]
CLEANUP: log: Return `size_t` from `sess_build_logline_orig()`
`sess_build_logline_orig()` takes a `size_t maxsize` as input and accordingly
should also return `size_t` instead of `int` as the resulting length. In
practice most of the callers already stored the result in a `size_t` anyways.
The few places that used an `int` were adjusted.
This Coccinelle patch was used to check for completeness:
@@
type T != size_t;
T var;
@@
(
* var = build_logline(...)
|
* var = build_logline_orig(...)
|
* var = sess_build_logline(...)
|
* var = sess_build_logline_orig(...)
)
Tim Duesterhus [Fri, 3 Apr 2026 21:28:56 +0000 (23:28 +0200)]
BUG/MINOR: log: Fix error message when using unavailable fetch in logfmt
The following configuration:
defaults
unique-id-format TEST-%[srv_name]
frontend fe_http
mode http
bind :::8080 v4v6
Emitted the following error:
[ALERT] (219835) : Parsing [./patch.cfg:2]: failed to parse unique-id : sample fetch <srv_name]> may not be reliably used here because it needs 'server' which is not available here.
The `]` in the name of the sample fetch should not be there.
This bug exists since at least HAProxy 2.4, which is the oldest supported
version. The fix should be backported there.
BUG/MINOR: quic: do not use hardcoded values in QMux TP frame builder
Reuse QUIC transport parameters value set in xprt_qstrm layer in frame
builder function. Prior to this patch, mux_quic would use different
values from the advertised ones.
MINOR: mux_quic: remove duplicate QMux local transport params
When QMux was first implemented, values used for emitted transport
parameters in xprt_qstrm and local flow control in mux_quic were
initialized separately. This is error prone in particular if a value is
change in one layer but not the other.
This patch fixes this by using xprt_qstrm_lparams() in QMux init
function. Mux flow control is then loaded with these values. Thus all
values are now initialized in a single place which is xprt_qstrm_init().
Tim Duesterhus [Sun, 12 Apr 2026 19:27:08 +0000 (21:27 +0200)]
CI: Merge `aws-lc.yml` and `aws-lc-fips.yml` into `aws-lc.yml`
These two jobs run on exactly the same triggers and are effectively variations
of each other. There is no need to have two separate workflows for them.
Tim Duesterhus [Sun, 12 Apr 2026 19:27:05 +0000 (21:27 +0200)]
CI: Fix regular expression escaping in matrix.py
This fixes:
.github/matrix.py:72: SyntaxWarning: "\." is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\\."? A raw string is also an option.
return re.match('^v[0-9]+(\.[0-9]+)*$', version_string)
.github/matrix.py:89: SyntaxWarning: "\." is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\\."? A raw string is also an option.
return re.match('^AWS-LC-FIPS-[0-9]+(\.[0-9]+)*$', version_string)
.github/matrix.py:106: SyntaxWarning: "\." is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\\."? A raw string is also an option.
return re.match('^v[0-9]+(\.[0-9]+)*-stable$', version_string)
Miroslav Zagorac [Fri, 20 Jan 2023 23:44:13 +0000 (00:44 +0100)]
MINOR: ot: renamed the variable dbg_indent_level to flt_ot_dbg_indent_level
The thread-local variable dbg_indent_level used a generic name that could
collide with identifiers in other compilation units. Renamed it to
flt_ot_dbg_indent_level so that it carried the flt_ot_ prefix consistent
with the rest of the OpenTracing filter namespace. The rename covered the
declaration, definition, and all macro references in debug.h, parser.c and
util.c.
Miroslav Zagorac [Thu, 19 Jan 2023 08:50:56 +0000 (09:50 +0100)]
BUILD: ot: removed explicit include path when building opentracing filter
The -Iaddons/ot/include flag in OT_CFLAGS allowed source files to use a
bare #include "include.h", which was fragile because it depended on the
compiler search path. Removed that flag from the Makefile and changed
every source file under addons/ot/src/ to use the relative include path
../include/include.h instead. This made header resolution explicit and
consistent with standard addon conventions.
Miroslav Zagorac [Wed, 18 Jan 2023 18:33:21 +0000 (19:33 +0100)]
BUG/MINOR: ot: fixed wrong NULL check in flt_ot_parse_cfg_group()
After calling flt_ot_conf_group_init() and storing the result in
flt_ot_current_group, the code incorrectly checked flt_ot_current_config
for NULL instead of the newly assigned flt_ot_current_group. This meant
a failed group init was never detected and the error path was never taken.
Miroslav Zagorac [Wed, 18 Jan 2023 16:14:16 +0000 (17:14 +0100)]
BUG/MINOR: ot: removed dead code in flt_ot_parse_cfg_str()
The local variable str was declared but never assigned a value other than
NULL. The error-handling block that called flt_ot_conf_str_free(&str) on
it was therefore a no-op. Removed both the unused variable and the dead
cleanup path.
Miroslav Zagorac [Thu, 25 Sep 2025 22:13:12 +0000 (00:13 +0200)]
CLEANUP: ot: use the item API for the variables trees
In flt_ot_vars_scope_dump(), switched from cebu64_first()/cebu64_next() to
cebu64_imm_first()/cebu64_imm_next() for iterating the variable name trees.
Since this function only reads variables under a read lock, the immutable
traversal API is the correct choice. Also updated the container_of()
member from 'node' to 'name_node' to match the current struct var layout.
MINOR: otel: changed instrument attr to use sample expressions
Replaced the static key-value attribute storage in update-form instruments
with sample-evaluated attributes, matching the log-record attr change.
The 'attr' keyword now accepts a key and a HAProxy sample expression
evaluated at runtime.
The struct (conf.h) changed from otelc_kv/attr_len to a list of
flt_otel_conf_sample entries. The parser (parser.c) calls
flt_otel_parse_cfg_sample() with n=1 per attr keyword. At runtime
(event.c) each attribute is evaluated via flt_otel_sample_eval() and
added via flt_otel_sample_add_kv() to a bare flt_otel_scope_data_kv,
which is passed to the meter.
Updated documentation, debug macro and test configurations.
MINOR: otel: changed log-record attr to use sample expressions
Replaced the static key-value attribute storage in log-record with
sample-evaluated attributes. The 'attr' keyword now accepts a key and a
HAProxy sample expression evaluated at runtime, instead of a static string
value.
The struct (conf.h) changed from otelc_kv/attr_len to a list of
flt_otel_conf_sample entries. The parser (parser.c) calls
flt_otel_parse_cfg_sample() with n=1 per attr keyword. At runtime
(event.c) each attribute is evaluated via flt_otel_sample_eval() and added
via flt_otel_sample_add_kv() to a bare flt_otel_scope_data_kv, which is
passed to logger->log_span().
Updated documentation, debug macro and test configurations.
MINOR: otel: added flt_otel_sample_eval and exposed flt_otel_sample_add_kv
Factored the sample evaluation logic out of flt_otel_sample_add() into a
new flt_otel_sample_eval() function that evaluates a sample definition
into an otelc_value. Both the log-format path and the bare sample
expression path are handled, with a flag_native parameter controlling
native type preservation for single-expression samples.
flt_otel_sample_add() now calls flt_otel_sample_eval() and dispatches the
result.
Made flt_otel_sample_add_kv() non-static so callers outside util.c can
add key-value pairs directly to a bare flt_otel_scope_data_kv without
requiring the full flt_otel_scope_data structure.
Miroslav Zagorac [Wed, 25 Mar 2026 11:17:03 +0000 (12:17 +0100)]
DOC: otel: test: added speed test guide and benchmark results
The test directory gained a speed test guide (README-test-speed)
explaining how to run performance benchmarks at various rate-limit levels,
together with benchmark result files for the standalone, composite,
context-propagation, and frontend-backend test configurations.
Miroslav Zagorac [Sat, 21 Mar 2026 17:39:52 +0000 (18:39 +0100)]
DOC: otel: test: added test README-* files
Added README documentation for each test configuration (sa, cmp, ctx,
fe-be, empty, full) describing event coverage, signal usage, instrument
tables, span hierarchies and run instructions.
Miroslav Zagorac [Sun, 12 Apr 2026 10:07:17 +0000 (12:07 +0200)]
DOC: otel: added documentation
Added the full documentation set for the OpenTelemetry filter.
The main README served as the user-facing guide covering build
instructions, core OpenTelemetry concepts, the complete filter
configuration reference, usage examples with worked scenarios,
CLI commands, and known limitations.
Supplementary documents provided a detailed configuration guide with
worked examples (README-configuration), an internal C structure reference
for developers (README-conf), a function reference organized by source
file (README-func), an architecture and implementation review
(README-implementation), and miscellaneous notes (README-misc).
Miroslav Zagorac [Fri, 13 Mar 2026 21:12:49 +0000 (22:12 +0100)]
MINOR: otel: test: added full-event test config
Added the 'full' test configuration that exercises all 29 supported OTel
filter events with all three signal types (traces, metrics, logs). Every
instrument definition has a corresponding update.
Added "log-record" as the third OpenTelemetry signal alongside traces
(span) and metrics (instrument). This includes the
flt_otel_conf_log_record structure definition, parser keyword defines,
the otel-scope section parser with optional "id", "event", "span", and
"attr" keywords followed by sample fetch expressions or a log-format
string, init/free lifecycle, scope list wiring, log-format evaluation
in flt_otel_scope_run_instrument_record(), a test configuration example,
log-record span reference validation in flt_otel_check(), and logger
handle creation, startup, and teardown in the filter lifecycle.
Added the "instrument" keyword to otel-scope sections for recording metric
measurements alongside traces.
Introduced flt_otel_conf_instrument holding instrument type, description,
unit, sample expressions, and optional key-value attributes. The
supported synchronous integer-precision instrument types were counters,
histograms, up-down counters, and gauges.
Instruments followed a two-form design: a "create" form defined a new
instrument with its type and value expression, while an "update" form
recorded measurements against an existing instrument with per-scope
attributes.
Instrument creation was performed lazily at first use with HA_ATOMIC_CAS
to guarantee thread-safe one-time initialization. The configuration
check phase validated that every update-form had a matching create-form
definition and that create-form names were unique across all scopes.
The meter lifecycle was integrated into filter init and deinit, starting
the meter alongside the tracer and shutting it down during cleanup.
Miroslav Zagorac [Thu, 26 Feb 2026 10:09:33 +0000 (11:09 +0100)]
MINOR: otel: added span link support
Added span link support, allowing a span to reference other spans or
extracted contexts without establishing a parent relationship.
Introduced the flt_otel_conf_link structure and added a links list to
flt_otel_conf_span. The parser accepted both an inline syntax on the span
declaration line ("span <name> link <target>") and a standalone multi-
argument form ("link <span> ..."), each creating a conf_link entry
appended to the span's link list.
At runtime, each configured link name was resolved against the active
spans and extracted contexts in the runtime context. Resolved references
were collected into flt_otel_scope_data_link entries and passed to the C
wrapper add_link API during span creation.
Initialization, cleanup, and debug dump routines were added for the link
data structures at both configuration and runtime levels.
Miroslav Zagorac [Tue, 27 Jan 2026 12:02:59 +0000 (13:02 +0100)]
MINOR: otel: test: added test and benchmark suite for the OTel filter
Added a test suite under addons/otel/test/ for the OpenTelemetry filter.
Five scenarios exercise different filter capabilities: standalone (sa)
covers all hook points including idle-timeout heartbeats, metrics and log
records; compact (cmp) covers the full request/response lifecycle with
ACL-based error handling; context (ctx) tests explicit inject/extract
propagation through numbered context variables; frontend/backend (fe/be)
tests distributed tracing across two HAProxy instances; and empty tests
bare filter initialisation with no active scopes.
A performance benchmarking script (test-speed.sh) uses wrk to measure
throughput and latency at different rate-limit settings (100% through 0%,
disabled, and filter-off). Each scenario includes comprehensive YAML
exporter definitions covering OTLP file/gRPC/HTTP, ostream, memory,
Zipkin, and Elasticsearch backends.
MINOR: otel: added log-format support to the sample parser and runtime
Extended flt_otel_parse_cfg_sample() to accept log-format strings in
addition to bare sample expressions. Added lf_expr and lf_used fields
to flt_otel_conf_sample.
Extended flt_otel_sample_add() to evaluate log-format expressions when
lf_used was set.
Miroslav Zagorac [Sun, 12 Apr 2026 10:20:15 +0000 (12:20 +0200)]
MEDIUM: otel: added group action for rule-based scope execution
Added the "otel-group" action keyword that allows executing a named group
of OTel scopes from HAProxy TCP and HTTP action rule contexts.
The new group.c module registers the "otel-group" keyword for all four
action contexts (tcp-request, tcp-response, http-request, http-response)
and implements the action lifecycle callbacks.
The parser flt_otel_group_parse() accepts a filter ID and group ID as
arguments, duplicates them into the action rule's argument slots, and
wires up the check, action, and release callbacks.
The post-parse validator flt_otel_group_check() resolves the filter ID and
group ID string references into direct configuration pointers by searching
the proxy's filter list for a matching OTel filter and then looking up the
named group within that filter's configuration.
The action handler flt_otel_group_action() retrieves the filter and group
configuration from the resolved rule arguments, verifies the filter is
attached to the stream and not disabled, then iterates through all scopes
in the group and executes each via flt_otel_scope_run() with a shared
timestamp pair. This allows operators to trigger OTel instrumentation
conditionally from HAProxy rules, for example applying different tracing
scopes based on ACL conditions or request properties.
Miroslav Zagorac [Sun, 12 Apr 2026 09:31:30 +0000 (11:31 +0200)]
MEDIUM: otel: added CLI commands for runtime filter management
Added HAProxy CLI commands that allow runtime inspection and modification
of OTel filter settings without requiring a configuration reload.
The new cli.c module registers CLI keywords under the "otel" prefix and
implements the following commands: flt_otel_cli_parse_status() displays a
comprehensive status report of all OTel filter instances including filter
ID, proxy, disabled state, hard-error mode, logging state, rate limit,
analyzer bits, and SDK diagnostic message count;
flt_otel_cli_parse_disabled() enables or disables filtering across all
instances; flt_otel_cli_parse_option() toggles the hard-error mode that
controls whether errors disable the filter for a stream or are silently
ignored; flt_otel_cli_parse_logging() manages the logging state with
support for off, on, and dontlog-normal modes; flt_otel_cli_parse_rate()
adjusts the sampling rate limit as a floating-point percentage; and
flt_otel_cli_parse_debug() sets the debug verbosity level in debug builds.
All modifications are applied atomically across every OTel filter instance
in every proxy.
The CLI initialization is called from flt_otel_ops_init() during filter
startup via flt_otel_cli_init(), which registers the keyword table through
cli_register_kw().
Supporting changes include the FLT_OTEL_U32_FLOAT macro for converting the
internal uint32_t rate representation to a human-readable percentage, the
FLT_OTEL_PROXIES_LIST_START/END iteration macros for traversing all OTel
filter instances across the proxy list, and flt_otel_filters_dump() for
debug logging of filter instances.
Miroslav Zagorac [Wed, 11 Feb 2026 05:55:44 +0000 (06:55 +0100)]
MINOR: otel: added prefix-based variable scanning
Introduced an alternative variable scanning strategy that directly walked
the CEB tree of HAProxy's variable store instead of maintaining a separate
tracking buffer.
The Makefile auto-detected whether struct var carried a "name" member
by inspecting include/haproxy/vars-t.h and conditionally defined
USE_OTEL_VARS_NAME. When enabled, the tracking buffer (flt_otel_ctx) and
its callback type were compiled out and replaced by direct tree walks.
flt_otel_vars_unset() walked the CEB tree for the resolved scope, removed
every variable whose normalized name matched the given prefix followed by
a dot, and adjusted the variable accounting. flt_otel_vars_get()
performed the same prefix scan under a read lock, denormalized each
matching variable name back to its original OTel form, and assembled the
results into an otelc_text_map.
A helper flt_otel_vars_get_scope() was added to resolve scope name strings
("txn", "sess", "proc", "req", "res") to the corresponding HAProxy
variable store. The set path skipped the tracking buffer update when
prefix scanning was available.
Miroslav Zagorac [Sun, 12 Apr 2026 09:25:14 +0000 (11:25 +0200)]
MEDIUM: otel: added HAProxy variable storage for context propagation
Added support for storing OTel span context in HAProxy transaction
variables as an alternative to HTTP headers, enabled by the OTEL_USE_VARS
compile flag.
The new vars.c module implements variable-based context propagation
through the HAProxy variable subsystem. Variable names are constructed
from a configurable prefix and the OTel propagation key, with dots
normalized to underscores for HAProxy variable name compatibility
and denormalized back during retrieval. The module provides
flt_otel_var_register() to pre-register variables at parse time,
flt_otel_var_set() and flt_otel_vars_unset() to store and clear context
key-value pairs in the txn scope, flt_otel_vars_get() to collect all
variables matching a prefix into an otelc_text_map for context extraction,
and flt_otel_vars_dump() for debug logging of all OTel variables.
The inject/extract keywords in the scope parser now accept an optional
"use-vars" argument alongside "use-headers", controlled by the new
FLT_OTEL_CTX_USE_VARS flag. Both storage types can be used simultaneously
on the same span context, allowing context to be propagated through both
HTTP headers and variables.
The scope runner in event.c was extended to handle variable-based
context in parallel with headers: during extraction, it reads matching
variables via flt_otel_vars_get() when FLT_OTEL_CTX_USE_VARS is set;
during injection, it stores each propagation key as a variable via
flt_otel_var_set(). The unused resource cleanup now also unsets context
variables when removing failed extraction contexts.
The filter attach callback registers and sets the sess.otel.uuid variable
with the generated session UUID, making the trace identifier available to
HAProxy log formats and ACL expressions.
The feature is conditionally compiled: the OTEL_USE_VARS flag controls
whether vars.c is included in the build and whether the "use-vars" keyword
is available in the configuration parser.
Miroslav Zagorac [Sun, 12 Apr 2026 09:20:59 +0000 (11:20 +0200)]
MEDIUM: otel: added HTTP header operations for context propagation
Added the HTTP header manipulation layer that enables span context
injection into and extraction from HAProxy's HTX message buffers,
completing the end-to-end context propagation path.
The new http.c module implements three public functions:
flt_otel_http_headers_get() extracts HTTP headers matching a name prefix
from the channel's HTX buffer into an otelc_text_map structure, stripping
the prefix and separator dash from header names before storage;
flt_otel_http_header_set() constructs a full header name from a prefix and
suffix joined by a dash, removes all existing occurrences, and optionally
adds the header with a new value; and flt_otel_http_headers_remove()
removes all headers matching a given prefix. A debug-only
flt_otel_http_headers_dump() logs all HTTP headers from a channel at
NOTICE level.
The scope runner in event.c now extracts propagation contexts from HTTP
headers before processing spans: for each configured extract context, it
calls flt_otel_http_headers_get() to read matching headers into a text
map, then passes the text map to flt_otel_scope_context_init() which
extracts the OTel span context from the carrier. After span execution,
the span runner injects the span context back into HTTP headers via
flt_otel_inject_http_headers() followed by flt_otel_http_header_set()
for each propagation key.
The unused resource cleanup in flt_otel_scope_free_unused() now also
removes contexts that failed extraction by deleting their associated
HTTP headers via flt_otel_http_headers_remove() before freeing the scope
context structure.
Miroslav Zagorac [Sun, 12 Apr 2026 09:17:11 +0000 (11:17 +0200)]
MEDIUM: otel: added context propagation via carrier interfaces
Added the span context injection and extraction layer that bridges the
OTel C wrapper's propagation API with HAProxy's HTTP headers and text map
carriers.
The new otelc.c module implements four public functions that wrap the
OTel C wrapper's context propagation methods: flt_otel_inject_text_map()
and flt_otel_inject_http_headers() serialize a span's context into
a text map or HTTP headers carrier for outbound propagation, while
flt_otel_extract_text_map() and flt_otel_extract_http_headers()
deserialize an inbound carrier into an otelc_span_context for parent
linking.
Each direction uses a pair of callbacks registered on the carrier
structure. The injection writers (flt_otel_text_map_writer_set_cb and
flt_otel_http_headers_writer_set_cb) store key-value pairs emitted by the
SDK into the carrier's text map via OTELC_TEXT_MAP_ADD(). The extraction
readers (flt_otel_text_map_reader_foreach_key_cb and
flt_otel_http_headers_reader_foreach_key_cb) iterate the carrier's text
map entries and pass each pair to the SDK's handler callback.
The scope context initialization in flt_otel_scope_context_init() now
calls flt_otel_extract_http_headers() to extract the span context from the
provided text map carrier and stores it in the scope context structure,
making extracted contexts available for parent linking in subsequent span
creation.
Miroslav Zagorac [Sun, 12 Apr 2026 08:58:38 +0000 (10:58 +0200)]
MEDIUM: otel: implemented scope execution and span management
Implemented the scope execution engine that creates OTel spans, evaluates
sample expressions to collect telemetry data, and manages span lifecycle
during request and response processing.
The scope runner flt_otel_scope_run() was expanded from a stub into a
complete implementation that evaluates ACL conditions on the scope,
extracts span contexts from HTTP headers when configured, iterates over
the scope's span definitions calling flt_otel_scope_run_span() for each,
marks and finishes completed spans, and cleans up unused runtime
resources.
The span runner flt_otel_scope_run_span() creates OTel spans via the
tracer with optional parent references (from other spans or extracted
contexts), collects telemetry by calling flt_otel_sample_add() for each
configured attribute, event, baggage and status entry, then applies the
collected data to the span (attributes, events with their own key-value
arrays, baggage items, and status code with description) and injects the
span context into HTTP headers when configured.
The sample evaluation layer converts HAProxy sample expressions into OTel
telemetry data. flt_otel_sample_add() evaluates each sample expression
against the stream, converts the result via flt_otel_sample_to_value()
which preserves native types (booleans as OTELC_VALUE_BOOL, integers as
OTELC_VALUE_INT64, all others as strings), and routes the key-value pair
to the appropriate collector based on the sample type (attribute, event,
baggage, or status). The key-value arrays grow dynamically using the
FLT_OTEL_ATTR_INIT_SIZE and FLT_OTEL_ATTR_INC_SIZE constants.
Span finishing is handled in two phases: flt_otel_scope_finish_mark()
marks spans and contexts for completion using exact name matching or
wildcards ("*" for all, "*req*" for request-direction, "*res*" for
response-direction), and flt_otel_scope_finish_marked() ends all marked
spans with a common monotonic timestamp and destroys their contexts.
Miroslav Zagorac [Sun, 12 Apr 2026 08:52:53 +0000 (10:52 +0200)]
MEDIUM: otel: wired OTel C wrapper library integration
Connected the OpenTelemetry C wrapper library to the filter lifecycle by
implementing the library initialization, tracer creation, memory and
thread callbacks, shutdown sequence, and span completion.
The flt_otel_lib_init() function now verifies the C wrapper library
version against the compiled headers, calls otelc_init() with the absolute
configuration file path, and creates the tracer via otelc_tracer_create().
On success, it registers HAProxy pool-based memory callbacks
(flt_otel_mem_malloc, flt_otel_mem_free) and a thread ID callback
(flt_otel_thread_id) through otelc_ext_init(), so the C++ SDK allocates
span and context objects from pool_head_otel_span_context. A custom log
handler (flt_otel_log_handler_cb) is registered via otelc_log_set_handler()
to count OTel SDK internal diagnostic messages in the flt_otel_drop_cnt
counter.
The per-thread init callback now starts the tracer thread via
OTELC_OPS(tracer, start) instead of unconditionally returning success.
The deinit callback saves the tracer handle before freeing the
configuration, then shuts down the library via otelc_deinit() after the
pool is destroyed, ensuring the ext callbacks remain valid while the
configuration structures are still being freed. In debug builds, it logs
wrapper statistics, attach counters, and per-event HTX usage counters
before shutdown.
The runtime context cleanup in flt_otel_runtime_context_free() now ends
all active spans with a common monotonic timestamp via
OTELC_OPSR(span, end_with_options) before freeing them. The scope context
cleanup in flt_otel_scope_context_free() now destroys the underlying OTel
span context via OTELC_OPSR(context, destroy).
The parser gained static storage for the debug memory tracker
(OTELC_DBG_MEM) and its initialization in the parse entry point, used when
compiled with the OTELC_DBG_MEM flag.
Miroslav Zagorac [Sun, 12 Apr 2026 08:30:03 +0000 (10:30 +0200)]
MEDIUM: otel: implemented filter callbacks and event dispatcher
Replaced the stub filter callbacks with full implementations that dispatch
OTel events through the scope execution engine, and added the supporting
debug, error handling and utility infrastructure.
The filter lifecycle callbacks (init, deinit, init_per_thread) now
initialize the OpenTelemetry C wrapper library, create the tracer from the
instrumentation configuration file, enable HTX stream filtering, and clean
up the configuration and memory pools on shutdown.
The stream callbacks (attach, stream_start, stream_set_backend,
stream_stop, detach, check_timeouts) create the per-stream runtime context
on attach with rate-limit based sampling, fire the corresponding OTel
events (on-stream-start, on-backend-set, on-stream-stop), manage the
idle timeout timer with reschedule logic in detach, and free the runtime
context in check_timeouts. The attach callback also registers the
required pre and post channel analyzers from the instrumentation
configuration.
The channel callbacks (start_analyze, pre_analyze, post_analyze,
end_analyze) register per-channel analyzers, map analyzer bits to event
indices via flt_otel_get_event(), and dispatch the matching events.
The end_analyze callback also fires the on-server-unavailable event
when response analyzers were configured but never executed.
The HTTP callbacks (http_headers, http_end, http_reply, and the debug-only
http_payload and http_reset) dispatch their respective request/response
events based on the channel direction.
The event dispatcher flt_otel_event_run() in event.c iterates over all
scopes matching a given event index and calls flt_otel_scope_run() for
each, sharing a common monotonic and wall-clock timestamp across all spans
within a single event.
Error handling is centralized in flt_otel_return_int() and
flt_otel_return_void(), which implement the hard-error/soft-error policy:
hard errors disable the filter for the stream, soft errors are silently
cleared.
The new debug.h header provides conditional debug macros
(FLT_OTEL_DBG_ARGS, FLT_OTEL_DBG_BUF) and the FLT_OTEL_LOG macro for
structured logging through the instrumentation's log server list. The
utility layer gained debug-only label functions for channel direction,
proxy mode, stream position, filter type, and analyzer bit name lookups.
projects / thirdparty / haproxy.git / log
