4.3. "otel-scope" section
4.4. "otel-group" section
5. Examples
+ 5.1. Benchmarking results
6. OTel CLI
7. Known bugs and limitations
configuration. If this is not done, the OTel filter in no way participates in
the work of HAProxy.
+As for the impact on HAProxy speed, this is documented with test results located
+in the test directory (see section 5.1). The speed of operation depends on the
+way the filter is used and the complexity of the configuration. In typical
+production use with a rate limit of 10% or less, the performance impact should
+be negligible (see the 'rate-limit' keyword).
+
The OTel filter allows intensive use of ACLs, which can be defined anywhere in
the configuration. Thus, it is possible to use the filter only for those
connections that are of interest to us.
% ./run-fe-be.sh
+5.1. Benchmarking results
+--------------------------
+
+To check the performance impact of the OTel filter, test configurations located
+in the test directory have been benchmarked. The test results (with the names
+README-speed-xxx, where xxx is the name of the configuration being tested) are
+also in the test directory.
+
+Testing was done with the wrk utility using 8 threads, 8 connections and a
+5-minute test duration. Detailed results and methodology are documented in the
+test/README-test-speed file.
+
+Below is a summary of the 'sa' (standalone) configuration results, which uses
+all possible events and demonstrates the worst-case scenario:
+
+ ---------------------------------------------------------------
+ rate-limit req/s avg latency overhead
+ ---------------------------------------------------------------
+ 100.0% 38,202 213.08 us 21.6%
+ 50.0% 42,777 190.49 us 12.2%
+ 25.0% 45,302 180.46 us 7.0%
+ 10.0% 46,879 174.69 us 3.7%
+ 2.5% 47,993 170.58 us 1.4%
+ disabled 48,788 167.74 us ~0
+ off 48,697 168.00 us baseline
+ ---------------------------------------------------------------
+
+The 'off' baseline is measured with the 'filter opentelemetry' directive
+commented out, so the filter is not loaded at all. The 'disabled' level has
+the filter loaded but disabled via 'option disabled', so the initialization
+overhead is included but no events are processed.
+
+As the table shows, with the rate limit set to 25% the overhead is about 7%.
+At 10% the overhead drops to 3.7%. In typical production use with a rate limit
+of 10% or less, the performance impact should be negligible.
+
+
6. OTel CLI
------------
--- /dev/null
+--- rate-limit 100.0 --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 182.58us 129.11us 16.19ms 98.11%
+ Req/Sec 5.63k 240.83 6.29k 69.74%
+ Latency Distribution
+ 50% 169.00us
+ 75% 183.00us
+ 90% 209.00us
+ 99% 367.00us
+ 13438310 requests in 5.00m, 3.24GB read
+Requests/sec: 44779.51
+Transfer/sec: 11.06MB
+----------------------------------------------------------------------
+
+--- rate-limit 75.0 --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 180.10us 122.51us 14.57ms 98.00%
+ Req/Sec 5.70k 253.08 6.28k 70.41%
+ Latency Distribution
+ 50% 169.00us
+ 75% 184.00us
+ 90% 206.00us
+ 99% 362.00us
+ 13613023 requests in 5.00m, 3.28GB read
+Requests/sec: 45361.63
+Transfer/sec: 11.20MB
+----------------------------------------------------------------------
+
+--- rate-limit 50.0 --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 176.59us 125.34us 15.58ms 98.02%
+ Req/Sec 5.81k 230.84 6.42k 72.14%
+ Latency Distribution
+ 50% 166.00us
+ 75% 182.00us
+ 90% 202.00us
+ 99% 361.00us
+ 13888448 requests in 5.00m, 3.35GB read
+Requests/sec: 46279.45
+Transfer/sec: 11.43MB
+----------------------------------------------------------------------
+
+--- rate-limit 25.0 --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 173.57us 118.35us 13.12ms 97.91%
+ Req/Sec 5.91k 257.69 6.46k 66.83%
+ Latency Distribution
+ 50% 162.00us
+ 75% 178.00us
+ 90% 199.00us
+ 99% 362.00us
+ 14122906 requests in 5.00m, 3.41GB read
+Requests/sec: 47060.71
+Transfer/sec: 11.62MB
+----------------------------------------------------------------------
+
+--- rate-limit 10.0 --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 170.85us 112.24us 10.72ms 97.84%
+ Req/Sec 6.00k 269.81 8.10k 69.46%
+ Latency Distribution
+ 50% 159.00us
+ 75% 172.00us
+ 90% 194.00us
+ 99% 361.00us
+ 14342642 requests in 5.00m, 3.46GB read
+Requests/sec: 47792.96
+Transfer/sec: 11.80MB
+----------------------------------------------------------------------
+
+--- rate-limit 2.5 --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 169.11us 127.52us 16.90ms 98.06%
+ Req/Sec 6.08k 261.57 6.55k 67.30%
+ Latency Distribution
+ 50% 158.00us
+ 75% 168.00us
+ 90% 186.00us
+ 99% 367.00us
+ 14527714 requests in 5.00m, 3.50GB read
+Requests/sec: 48409.62
+Transfer/sec: 11.96MB
+----------------------------------------------------------------------
+
+--- rate-limit 0.0 --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 168.09us 108.96us 9.07ms 97.81%
+ Req/Sec 6.10k 284.22 6.55k 70.65%
+ Latency Distribution
+ 50% 157.00us
+ 75% 167.00us
+ 90% 184.00us
+ 99% 362.00us
+ 14580762 requests in 5.00m, 3.52GB read
+Requests/sec: 48586.42
+Transfer/sec: 12.00MB
+----------------------------------------------------------------------
+
+--- rate-limit disabled --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 168.57us 118.05us 13.51ms 97.94%
+ Req/Sec 6.09k 251.47 6.99k 67.33%
+ Latency Distribution
+ 50% 158.00us
+ 75% 167.00us
+ 90% 184.00us
+ 99% 363.00us
+ 14557824 requests in 5.00m, 3.51GB read
+Requests/sec: 48509.96
+Transfer/sec: 11.98MB
+----------------------------------------------------------------------
+
+--- rate-limit off --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 168.64us 120.15us 14.49ms 98.01%
+ Req/Sec 6.09k 267.94 6.57k 66.19%
+ Latency Distribution
+ 50% 158.00us
+ 75% 167.00us
+ 90% 184.00us
+ 99% 361.00us
+ 14551312 requests in 5.00m, 3.51GB read
+Requests/sec: 48488.23
+Transfer/sec: 11.98MB
+----------------------------------------------------------------------
+
--- /dev/null
+--- rate-limit 100.0 --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 270.35us 136.92us 16.02ms 97.98%
+ Req/Sec 3.77k 217.57 5.33k 67.74%
+ Latency Distribution
+ 50% 264.00us
+ 75% 287.00us
+ 90% 309.00us
+ 99% 494.00us
+ 9012538 requests in 5.00m, 2.17GB read
+Requests/sec: 30031.85
+Transfer/sec: 7.42MB
+----------------------------------------------------------------------
+
+--- rate-limit 75.0 --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 242.56us 131.47us 9.82ms 94.77%
+ Req/Sec 4.21k 218.42 5.61k 68.12%
+ Latency Distribution
+ 50% 246.00us
+ 75% 279.00us
+ 90% 308.00us
+ 99% 464.00us
+ 10050409 requests in 5.00m, 2.42GB read
+Requests/sec: 33490.26
+Transfer/sec: 8.27MB
+----------------------------------------------------------------------
+
+--- rate-limit 50.0 --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 215.92us 130.82us 9.93ms 96.81%
+ Req/Sec 4.73k 243.84 7.13k 67.13%
+ Latency Distribution
+ 50% 208.00us
+ 75% 264.00us
+ 90% 300.00us
+ 99% 439.00us
+ 11307386 requests in 5.00m, 2.73GB read
+Requests/sec: 37678.82
+Transfer/sec: 9.31MB
+----------------------------------------------------------------------
+
+--- rate-limit 25.0 --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 192.36us 132.47us 13.75ms 96.79%
+ Req/Sec 5.33k 260.46 6.17k 66.30%
+ Latency Distribution
+ 50% 166.00us
+ 75% 227.00us
+ 90% 280.00us
+ 99% 407.00us
+ 12734770 requests in 5.00m, 3.07GB read
+Requests/sec: 42448.91
+Transfer/sec: 10.48MB
+----------------------------------------------------------------------
+
+--- rate-limit 10.0 --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 180.08us 127.35us 13.34ms 97.06%
+ Req/Sec 5.71k 272.98 6.40k 67.94%
+ Latency Distribution
+ 50% 161.00us
+ 75% 183.00us
+ 90% 250.00us
+ 99% 386.00us
+ 13641901 requests in 5.00m, 3.29GB read
+Requests/sec: 45457.92
+Transfer/sec: 11.23MB
+----------------------------------------------------------------------
+
+--- rate-limit 2.5 --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 171.64us 107.08us 5.69ms 96.57%
+ Req/Sec 5.97k 289.99 6.55k 68.53%
+ Latency Distribution
+ 50% 159.00us
+ 75% 171.00us
+ 90% 195.00us
+ 99% 372.00us
+ 14268464 requests in 5.00m, 3.44GB read
+Requests/sec: 47545.77
+Transfer/sec: 11.74MB
+----------------------------------------------------------------------
+
+--- rate-limit 0.0 --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 168.86us 104.53us 5.75ms 97.73%
+ Req/Sec 6.07k 282.19 6.59k 67.47%
+ Latency Distribution
+ 50% 158.00us
+ 75% 168.00us
+ 90% 186.00us
+ 99% 361.00us
+ 14498699 requests in 5.00m, 3.50GB read
+Requests/sec: 48312.96
+Transfer/sec: 11.93MB
+----------------------------------------------------------------------
+
+--- rate-limit disabled --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 168.36us 129.52us 17.68ms 98.13%
+ Req/Sec 6.11k 263.70 6.83k 70.42%
+ Latency Distribution
+ 50% 157.00us
+ 75% 167.00us
+ 90% 183.00us
+ 99% 363.00us
+ 14590953 requests in 5.00m, 3.52GB read
+Requests/sec: 48620.36
+Transfer/sec: 12.01MB
+----------------------------------------------------------------------
+
+--- rate-limit off --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 168.73us 120.51us 15.03ms 98.02%
+ Req/Sec 6.09k 270.88 6.55k 68.99%
+ Latency Distribution
+ 50% 158.00us
+ 75% 167.00us
+ 90% 185.00us
+ 99% 360.00us
+ 14538507 requests in 5.00m, 3.51GB read
+Requests/sec: 48445.53
+Transfer/sec: 11.97MB
+----------------------------------------------------------------------
+
--- /dev/null
+--- rate-limit 100.0 --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 238.75us 129.45us 15.59ms 98.45%
+ Req/Sec 4.27k 75.22 5.28k 77.45%
+ Latency Distribution
+ 50% 228.00us
+ 75% 243.00us
+ 90% 262.00us
+ 99% 410.00us
+ 10206938 requests in 5.00m, 2.46GB read
+Requests/sec: 34011.80
+Transfer/sec: 8.40MB
+----------------------------------------------------------------------
+
+--- rate-limit 75.0 --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 230.08us 201.50us 32.12ms 99.25%
+ Req/Sec 4.46k 83.43 5.36k 75.61%
+ Latency Distribution
+ 50% 222.00us
+ 75% 241.00us
+ 90% 261.00us
+ 99% 401.00us
+ 10641998 requests in 5.00m, 2.57GB read
+Requests/sec: 35461.59
+Transfer/sec: 8.76MB
+----------------------------------------------------------------------
+
+--- rate-limit 50.0 --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 222.33us 242.52us 35.13ms 99.43%
+ Req/Sec 4.62k 99.86 5.78k 79.00%
+ Latency Distribution
+ 50% 211.00us
+ 75% 237.00us
+ 90% 259.00us
+ 99% 400.00us
+ 11046951 requests in 5.00m, 2.66GB read
+Requests/sec: 36810.91
+Transfer/sec: 9.09MB
+----------------------------------------------------------------------
+
+--- rate-limit 25.0 --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 210.95us 117.20us 10.57ms 97.85%
+ Req/Sec 4.84k 101.85 6.04k 68.10%
+ Latency Distribution
+ 50% 198.00us
+ 75% 222.00us
+ 90% 252.00us
+ 99% 394.00us
+ 11551741 requests in 5.00m, 2.79GB read
+Requests/sec: 38493.03
+Transfer/sec: 9.51MB
+----------------------------------------------------------------------
+
+--- rate-limit 10.0 --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 206.15us 209.34us 31.73ms 99.27%
+ Req/Sec 4.99k 112.62 5.36k 71.21%
+ Latency Distribution
+ 50% 193.00us
+ 75% 210.00us
+ 90% 237.00us
+ 99% 387.00us
+ 11924489 requests in 5.00m, 2.88GB read
+Requests/sec: 39735.09
+Transfer/sec: 9.81MB
+----------------------------------------------------------------------
+
+--- rate-limit 2.5 --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 202.35us 180.56us 27.28ms 99.10%
+ Req/Sec 5.08k 145.06 8.27k 71.24%
+ Latency Distribution
+ 50% 191.00us
+ 75% 205.00us
+ 90% 223.00us
+ 99% 374.00us
+ 12131047 requests in 5.00m, 2.93GB read
+Requests/sec: 40423.43
+Transfer/sec: 9.98MB
+----------------------------------------------------------------------
+
+--- rate-limit 0.0 --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 200.88us 223.19us 32.70ms 99.44%
+ Req/Sec 5.13k 151.03 6.55k 69.46%
+ Latency Distribution
+ 50% 190.00us
+ 75% 203.00us
+ 90% 218.00us
+ 99% 367.00us
+ 12256706 requests in 5.00m, 2.96GB read
+Requests/sec: 40842.16
+Transfer/sec: 10.09MB
+----------------------------------------------------------------------
+
+--- rate-limit disabled --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 194.91us 222.55us 33.80ms 99.47%
+ Req/Sec 5.29k 167.52 5.95k 68.32%
+ Latency Distribution
+ 50% 184.00us
+ 75% 197.00us
+ 90% 214.00us
+ 99% 353.00us
+ 12633928 requests in 5.00m, 3.05GB read
+Requests/sec: 42112.54
+Transfer/sec: 10.40MB
+----------------------------------------------------------------------
+
+--- rate-limit off --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 194.37us 166.76us 28.75ms 99.09%
+ Req/Sec 5.28k 160.02 5.86k 68.02%
+ Latency Distribution
+ 50% 184.00us
+ 75% 197.00us
+ 90% 214.00us
+ 99% 355.00us
+ 12622896 requests in 5.00m, 3.04GB read
+Requests/sec: 42062.31
+Transfer/sec: 10.39MB
+----------------------------------------------------------------------
+
--- /dev/null
+--- rate-limit 100.0 --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 213.08us 136.99us 17.58ms 98.10%
+ Req/Sec 4.80k 251.04 5.97k 68.01%
+ Latency Distribution
+ 50% 203.00us
+ 75% 223.00us
+ 90% 245.00us
+ 99% 405.00us
+ 11464278 requests in 5.00m, 2.77GB read
+Requests/sec: 38201.61
+Transfer/sec: 9.44MB
+----------------------------------------------------------------------
+
+--- rate-limit 75.0 --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 202.18us 121.42us 12.04ms 97.72%
+ Req/Sec 5.05k 248.48 5.85k 65.69%
+ Latency Distribution
+ 50% 194.00us
+ 75% 219.00us
+ 90% 245.00us
+ 99% 393.00us
+ 12071015 requests in 5.00m, 2.91GB read
+Requests/sec: 40223.31
+Transfer/sec: 9.94MB
+----------------------------------------------------------------------
+
+--- rate-limit 50.0 --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 190.49us 117.32us 7.33ms 97.62%
+ Req/Sec 5.37k 265.60 6.98k 65.98%
+ Latency Distribution
+ 50% 181.00us
+ 75% 208.00us
+ 90% 237.00us
+ 99% 383.00us
+ 12837427 requests in 5.00m, 3.10GB read
+Requests/sec: 42777.17
+Transfer/sec: 10.57MB
+----------------------------------------------------------------------
+
+--- rate-limit 25.0 --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 180.46us 123.40us 13.20ms 97.80%
+ Req/Sec 5.69k 242.93 7.75k 68.66%
+ Latency Distribution
+ 50% 165.00us
+ 75% 194.00us
+ 90% 223.00us
+ 99% 375.00us
+ 13595213 requests in 5.00m, 3.28GB read
+Requests/sec: 45302.34
+Transfer/sec: 11.19MB
+----------------------------------------------------------------------
+
+--- rate-limit 10.0 --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 174.69us 129.17us 16.50ms 97.93%
+ Req/Sec 5.89k 260.40 7.03k 69.57%
+ Latency Distribution
+ 50% 160.00us
+ 75% 178.00us
+ 90% 210.00us
+ 99% 374.00us
+ 14068388 requests in 5.00m, 3.39GB read
+Requests/sec: 46879.07
+Transfer/sec: 11.58MB
+----------------------------------------------------------------------
+
+--- rate-limit 2.5 --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 170.58us 116.89us 11.49ms 97.74%
+ Req/Sec 6.03k 249.35 6.54k 67.44%
+ Latency Distribution
+ 50% 158.00us
+ 75% 170.00us
+ 90% 192.00us
+ 99% 375.00us
+ 14402604 requests in 5.00m, 3.47GB read
+Requests/sec: 47992.71
+Transfer/sec: 11.85MB
+----------------------------------------------------------------------
+
+--- rate-limit 0.0 --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 167.96us 114.36us 10.99ms 97.81%
+ Req/Sec 6.12k 266.16 6.57k 70.70%
+ Latency Distribution
+ 50% 157.00us
+ 75% 166.00us
+ 90% 183.00us
+ 99% 370.00us
+ 14622790 requests in 5.00m, 3.53GB read
+Requests/sec: 48726.40
+Transfer/sec: 12.04MB
+----------------------------------------------------------------------
+
+--- rate-limit disabled --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 167.74us 114.11us 11.10ms 97.82%
+ Req/Sec 6.13k 251.71 6.57k 69.59%
+ Latency Distribution
+ 50% 157.00us
+ 75% 166.00us
+ 90% 182.00us
+ 99% 368.00us
+ 14641307 requests in 5.00m, 3.53GB read
+Requests/sec: 48788.18
+Transfer/sec: 12.05MB
+----------------------------------------------------------------------
+
+--- rate-limit off --------------------------------------------------
+Running 5m test @ http://localhost:10080/index.html
+ 8 threads and 8 connections
+ Thread Stats Avg Stdev Max +/- Stdev
+ Latency 168.00us 112.83us 11.07ms 97.79%
+ Req/Sec 6.12k 264.12 7.23k 68.95%
+ Latency Distribution
+ 50% 157.00us
+ 75% 166.00us
+ 90% 183.00us
+ 99% 369.00us
+ 14613970 requests in 5.00m, 3.53GB read
+Requests/sec: 48697.01
+Transfer/sec: 12.03MB
+----------------------------------------------------------------------
+
--- /dev/null
+ -----------------------------------------
+ HAProxy OTEL filter speed test guide
+ Version 1.0
+ ( Last update: 2026-04-09 )
+ -----------------------------------------
+ Author : Miroslav Zagorac
+ Contact : mzagorac at haproxy dot com
+
+
+SUMMARY
+--------
+
+ 1. Overview
+ 2. Prerequisites
+ 3. Running the test
+ 4. Test parameters
+ 5. Rate-limit levels
+ 6. Test configurations
+ 7. Results
+ 7.1. Standalone (sa)
+ 7.2. Comparison (cmp)
+ 7.3. Context propagation (ctx)
+ 7.4. Frontend / backend (fe-be)
+ 8. Summary
+
+
+1. Overview
+------------
+
+The test-speed.sh script measures the performance impact of the OTEL filter on
+HAProxy at various rate-limit settings. For each test configuration, the script
+iterates through a series of rate-limit values -- from full tracing (100%) down
+to the filter being completely removed -- measuring throughput and latency at
+each level.
+
+The script uses template files (haproxy.cfg.in and otel.cfg.in) from each test
+directory to generate the actual configuration files. A sed substitution
+adjusts the rate-limit value (or disables/removes the filter) before each run.
+
+
+2. Prerequisites
+-----------------
+
+The following tools must be installed and available in PATH:
+
+ - thttpd : a lightweight HTTP server used as the backend origin server.
+ It serves a small static HTML file (index.html) on port 8000.
+
+ - wrk : an HTTP benchmarking tool that generates the test load.
+ See https://github.com/wg/wrk
+
+
+3. Running the test
+--------------------
+
+The test is executed from the test directory. It can be run for all
+configurations at once or for a single configuration.
+
+To run all configurations:
+
+ % ./test-speed.sh all
+
+This produces result files in the _logs directory:
+
+ _logs/README-speed-fe-be
+ _logs/README-speed-sa
+ _logs/README-speed-cmp
+ _logs/README-speed-ctx
+
+To run a single configuration:
+
+ % ./test-speed.sh <cfg> [<dir>]
+
+Where <cfg> corresponds to a run-<cfg>.sh script and <dir> is the configuration
+directory (defaults to <cfg>). For example:
+
+ % ./test-speed.sh sa
+ % ./test-speed.sh fe-be fe
+ % ./test-speed.sh cmp
+ % ./test-speed.sh ctx
+
+
+4. Test parameters
+-------------------
+
+The wrk benchmarking tool is invoked with the following parameters:
+
+ -t8 8 threads
+ -c8 8 concurrent connections
+ -d300 5-minute test duration (300 seconds)
+ --latency latency distribution reporting
+
+Each rate-limit level is tested sequentially. Between runs, HAProxy is stopped
+via SIGUSR1 and restarted with the next rate-limit configuration. A 10-second
+pause separates consecutive runs.
+
+The backend origin server (thttpd) serves a small static HTML page (index.html,
+approximately 50 bytes) on port 8000. HAProxy listens on port 10080 and proxies
+requests to the origin.
+
+
+5. Rate-limit levels
+---------------------
+
+The script tests nine rate-limit levels in the following order:
+
+ 100.0 - the filter processes every stream (worst case)
+ 75.0 - the filter processes 75% of streams
+ 50.0 - the filter processes 50% of streams
+ 25.0 - the filter processes 25% of streams
+ 10.0 - the filter processes 10% of streams
+ 2.5 - the filter processes 2.5% of streams
+ 0.0 - the filter is loaded and attached to every stream but never
+ processes any telemetry (the rate-limit check always fails);
+ this measures the per-stream attach and detach overhead
+
+ disabled - the filter is loaded but disabled via 'option disabled'; it is not
+ attached to streams at all; this measures the cost of loading and
+ initializing the filter library without any per-stream work
+
+ off - the 'filter opentelemetry' and 'otel-group' directives are
+ commented out of haproxy.cfg; the filter is not loaded and has zero
+ presence in the processing path; this is the absolute baseline
+
+In the result tables, the 'overhead' column is the throughput loss relative to
+the 'off' baseline, expressed as a percentage:
+
+ overhead = (req/s_off - req/s_test) / req/s_off * 100
+
+
+6. Test configurations
+-----------------------
+
+Four OTEL filter configurations are tested. They differ in complexity and in
+the features they exercise:
+
+ sa - Standalone. Uses all possible HAProxy filter events with spans,
+ attributes, events, links, baggage, status, metrics and groups.
+ This is the most comprehensive single-instance configuration and
+ represents the worst-case scenario.
+
+ cmp - Comparison. A simplified configuration made for comparison with
+ other tracing implementations. It uses a reduced span hierarchy
+ without context propagation, groups or metrics. This is closer to
+ a typical production deployment.
+
+ ctx - Context propagation. Similar to 'sa' in scope coverage, but spans
+ are opened using extracted span contexts (inject/extract via HAProxy
+ variables) as parent references instead of direct span names. This
+ adds the overhead of context serialization, variable storage and
+ deserialization on every scope execution.
+
+ fe-be - Frontend / backend. Two cascaded HAProxy instances: the frontend
+ (fe) creates the root trace and injects span context into HTTP
+ headers; the backend (be) extracts the context and continues the
+ trace. This configuration measures the combined overhead of two
+ OTEL filter instances plus the inter-process context propagation
+ cost.
+
+ Note: the rate-limit is varied only on the frontend. The backend
+ always runs with its default configuration (filter enabled,
+ hard-errors on). The backend configuration is only modified for
+ the 'disabled' and 'off' levels.
+
+
+7. Results
+-----------
+
+The tables below summarize the benchmarking results. The 'req/s' column shows
+the sustained request rate reported by wrk, 'avg latency' is the average
+response time, and 'overhead' is the throughput loss relative to the 'off'
+baseline.
+
+
+7.1. Standalone (sa)
+---------------------
+
+ ---------------------------------------------------------------
+ rate-limit req/s avg latency overhead
+ ---------------------------------------------------------------
+ 100.0% 38,202 213.08 us 21.6%
+ 75.0% 40,223 202.18 us 17.4%
+ 50.0% 42,777 190.49 us 12.2%
+ 25.0% 45,302 180.46 us 7.0%
+ 10.0% 46,879 174.69 us 3.7%
+ 2.5% 47,993 170.58 us 1.4%
+ 0.0% 48,726 167.96 us ~0
+ disabled 48,788 167.74 us ~0
+ off 48,697 168.00 us baseline
+ ---------------------------------------------------------------
+
+With all possible events active, the sa configuration at 100% rate-limit shows
+a 22% throughput reduction. At 10% rate-limit the overhead drops to 3.7%, and
+at 2.5% it is barely measurable at 1.4%. The 'disabled' and '0.0' levels show
+no measurable overhead above the baseline.
+
+
+7.2. Comparison (cmp)
+----------------------
+
+ ---------------------------------------------------------------
+ rate-limit req/s avg latency overhead
+ ---------------------------------------------------------------
+ 100.0% 44,780 182.58 us 7.6%
+ 75.0% 45,362 180.10 us 6.4%
+ 50.0% 46,279 176.59 us 4.6%
+ 25.0% 47,061 173.57 us 2.9%
+ 10.0% 47,793 170.85 us 1.4%
+ 2.5% 48,410 169.11 us 0.2%
+ 0.0% 48,586 168.09 us ~0
+ disabled 48,510 168.57 us ~0
+ off 48,488 168.64 us baseline
+ ---------------------------------------------------------------
+
+The simplified cmp configuration shows significantly lower overhead than sa at
+every rate-limit level. At 100% rate-limit the overhead is 7.6%, at 10% it is
+1.4%, and below 2.5% it becomes indistinguishable from the baseline.
+
+
+7.3. Context propagation (ctx)
+-------------------------------
+
+ ---------------------------------------------------------------
+ rate-limit req/s avg latency overhead
+ ---------------------------------------------------------------
+ 100.0% 30,032 270.35 us 38.0%
+ 75.0% 33,490 242.56 us 30.9%
+ 50.0% 37,679 215.92 us 22.2%
+ 25.0% 42,449 192.36 us 12.4%
+ 10.0% 45,458 180.08 us 6.2%
+ 2.5% 47,546 171.64 us 1.9%
+ 0.0% 48,313 168.86 us 0.3%
+ disabled 48,620 168.36 us ~0
+ off 48,446 168.73 us baseline
+ ---------------------------------------------------------------
+
+The ctx configuration is the most expensive due to the inject/extract cycle on
+every scope execution. At 100% rate-limit the overhead reaches 38%. However,
+the cost scales linearly with the rate-limit: at 10% the overhead is 6.2%, and
+at 2.5% it drops to 1.9%. The filter attachment overhead (0.0% vs off) is
+negligible at 0.3%.
+
+
+7.4. Frontend / backend (fe-be)
+--------------------------------
+
+ ---------------------------------------------------------------
+ rate-limit req/s avg latency overhead
+ ---------------------------------------------------------------
+ 100.0% 34,012 238.75 us 19.1%
+ 75.0% 35,462 230.08 us 15.7%
+ 50.0% 36,811 222.33 us 12.5%
+ 25.0% 38,493 210.95 us 8.5%
+ 10.0% 39,735 206.15 us 5.5%
+ 2.5% 40,423 202.35 us 3.9%
+ 0.0% 40,842 200.88 us 2.9%
+ disabled 42,113 194.91 us ~0
+ off 42,062 194.37 us baseline
+ ---------------------------------------------------------------
+
+The fe-be configuration involves two HAProxy instances in series, so the
+absolute baseline (off) is already lower at 42,062 req/s due to the extra
+network hop. The rate-limit is varied only on the frontend; the backend
+always has the filter loaded with hard-errors enabled.
+
+This explains the 2.9% overhead at rate-limit 0.0: even though the frontend
+never traces, the backend filter still attaches to every stream, attempts to
+extract context from the HTTP headers, fails (because the frontend did not
+inject any context), and the hard-errors option stops further processing.
+This per-stream attach/extract/error cycle accounts for the residual cost.
+
+When both instances have the filter disabled (disabled level), the overhead
+is within measurement noise, consistent with the single-instance
+configurations.
+
+
+8. Summary
+-----------
+
+The table below shows the overhead for each configuration at selected rate-limit
+levels:
+
+ ---------------------------------------------------
+ rate-limit sa cmp ctx fe-be
+ ---------------------------------------------------
+ 100.0% 21.6% 7.6% 38.0% 19.1%
+ 25.0% 7.0% 2.9% 12.4% 8.5%
+ 10.0% 3.7% 1.4% 6.2% 5.5%
+ 2.5% 1.4% 0.2% 1.9% 3.9%
+ ---------------------------------------------------
+
+Key observations:
+
+ - The overhead scales approximately linearly with the rate-limit value.
+ Reducing the rate-limit from 100% to 10% eliminates the vast majority
+ of the cost in all configurations.
+
+ - The cmp configuration, which uses a reduced span hierarchy typical of
+ production deployments, adds only 1.4% overhead at a 10% rate-limit.
+
+ - The sa configuration, which exercises all possible events, stays at about
+ 7% overhead at a 25% rate-limit and below 4% at 10%.
+
+ - The ctx configuration is the most expensive due to the inject/extract
+ context propagation on every scope. It is designed as a stress test for
+ the propagation mechanism rather than a practical production template.
+
+ - The fe-be configuration carries a higher fixed cost because two HAProxy
+ instances are involved and the backend filter processes context extraction
+ regardless of the frontend rate-limit setting.
+
+ - Loading the filter but disabling it via 'option disabled' adds no measurable
+ overhead in any configuration.
+
+ - The filter attachment cost without any telemetry processing (rate-limit 0.0)
+ is 0.3% or less for single-instance configurations (sa, cmp, ctx).
+
+ - In typical production use with a rate-limit of 10% or less, the performance
+ impact of the OTEL filter should be negligible for single-instance deployments.
projects / thirdparty / haproxy.git / commitdiff
