Various settings can be configured related to QUIC congestion controler.
This patch duplicates them to be able to set independent values on
frontend and backend sides.
As with previous patch, option are renamed to use "fe/be" unified
prefixes. This is part of the current serie of commits which unify QUIC
settings. Older options are deprecated and will be removed on 3.5
release.
- tune.pool-high-fd-ratio
- tune.pool-low-fd-ratio
- tune.pt.zero-copy-forwarding
+ - tune.quic.be.cc.cubic-min-losses
+ - tune.quic.be.cc.hystart
+ - tune.quic.be.cc.max-frame-loss
+ - tune.quic.be.cc.reorder-ratio
- tune.quic.be.sec.glitches-threshold
- - tune.quic.cc-hystart
- - tune.quic.cc.cubic.min-losses
+ - tune.quic.cc.cubic.min-losses (deprecated)
+ - tune.quic.cc-hystart (deprecated)
- tune.quic.disable-tx-pacing
- tune.quic.disable-udp-gso
+ - tune.quic.fe.cc.cubic-min-losses
+ - tune.quic.fe.cc.hystart
+ - tune.quic.fe.cc.max-frame-loss
+ - tune.quic.fe.cc.reorder-ratio
- tune.quic.fe.sec.glitches-threshold
- tune.quic.frontend.glitches-threshold (deprecated)
- tune.quic.frontend.max-data-size
- tune.quic.frontend.stream-data-ratio
- tune.quic.frontend.default-max-window-size
- tune.quic.listen
- - tune.quic.max-frame-loss
- - tune.quic.reorder-ratio
+ - tune.quic.max-frame-loss (deprecated)
+ - tune.quic.reorder-ratio (deprecated)
- tune.quic.retry-threshold
- tune.quic.socket-owner
- tune.quic.zero-copy-fwd-send
See also: tune.disable-zero-copy-forwarding, option splice-auto,
option splice-request and option splice-response
-tune.quic.cc-hystart { on | off }
- Enables ('on') or disabled ('off') the HyStart++ (RFC 9406) algorithm for
- QUIC connections used as a replacement for the slow start phase of congestion
- control algorithms which may cause high packet loss. It is disabled by default.
+tune.quic.disable-tx-pacing
+ Disables pacing support for QUIC emission. The purpose of pacing is to smooth
+ emission of data to reduce network losses. In most scenario, it will
+ significantly improve network throughput by avoiding retransmissions.
+ However, it can be useful to deactivate it for networks with very high
+ bandwidth/low latency characteristics to prevent unwanted delay and reduce
+ CPU consumption.
+
+ See also the "quic-cc-algo" bind option.
+
+tune.quic.disable-udp-gso
+ Disable UDP GSO emission. This kernel feature allows to emit multiple
+ datagrams via a single system call which is more efficient for large
+ transfer. It may be useful to disable it on developers suggestion when
+ suspecting an issue on emission.
-tune.quic.cc.cubic.min-losses <number>
+tune.quic.be.cc.cubic-min-losses <number>
+tune.quic.fe.cc.cubic-min-losses <number>
Defines how many lost packets are needed for the Cubic congestion control
algorithm to really consider a loss event. Normally, any loss event is
considered as the result of a congestion and is sufficient for Cubic to
compare some metrics. Never go beyond 2 without an expert's prior analysis of
the situation. The default and minimum value is 1. Always use 1.
-tune.quic.disable-tx-pacing
- Disables pacing support for QUIC emission. The purpose of pacing is to smooth
- emission of data to reduce network losses. In most scenario, it will
- significantly improve network throughput by avoiding retransmissions.
- However, it can be useful to deactivate it for networks with very high
- bandwidth/low latency characteristics to prevent unwanted delay and reduce
- CPU consumption.
+tune.quic.cc.cubic.min-losses <number> (deprecated)
+ This keyword has been deprecated in 3.3 and will be removed in 3.5. It is
+ part of the streamlining process apply on QUIC configuration. If used, this
+ setting will only be applied on frontend connections.
- See also the "quic-cc-algo" bind option.
+tune.quic.be.cc.hystart { on | off }
+tune.quic.fe.cc.hystart { on | off }
+ Enables ('on') or disabled ('off') the HyStart++ (RFC 9406) algorithm for
+ QUIC connections used as a replacement for the slow start phase of congestion
+ control algorithms which may cause high packet loss. It is disabled by default.
-tune.quic.disable-udp-gso
- Disable UDP GSO emission. This kernel feature allows to emit multiple
- datagrams via a single system call which is more efficient for large
- transfer. It may be useful to disable it on developers suggestion when
- suspecting an issue on emission.
+tune.quic.cc-hystart { on | off } (deprecated)
+ This keyword has been deprecated in 3.3 and will be removed in 3.5. It is
+ part of the streamlining process apply on QUIC configuration. If used, this
+ setting will only be applied on frontend connections.
+
+tune.quic.be.cc.max-frame-loss <number>
+tune.quic.fe.cc.max-frame-loss <number>
+ Sets the limit for which a single QUIC frame can be marked as lost. If
+ exceeded, the connection is considered as failing and is closed immediately.
+
+ The default value is 10.
+
+tune.quic.max-frame-loss <number> (deprecated)
+ This keyword has been deprecated in 3.3 and will be removed in 3.5. It is
+ part of the streamlining process apply on QUIC configuration. If used, this
+ setting will only be applied on frontend connections.
+
+tune.quic.be.cc.reorder-ratio <0..100, in percent>
+tune.quic.fe.cc.reorder-ratio <0..100, in percent>
+ The ratio applied to the packet reordering threshold calculated. It may
+ trigger a high packet loss detection when too small.
+
+ The default value is 50.
+
+tune.quic.reorder-ratio <0..100, in percent> (deprecated)
+ This keyword has been deprecated in 3.3 and will be removed in 3.5. It is
+ part of the streamlining process apply on QUIC configuration. If used, this
+ setting will only be applied on frontend connections.
tune.quic.be.sec.glitches-threshold <number>
tune.quic.fe.sec.glitches-threshold <number>
See also "quic_enabled" sample fetch.
-tune.quic.max-frame-loss <number>
- Sets the limit for which a single QUIC frame can be marked as lost. If
- exceeded, the connection is considered as failing and is closed immediately.
-
- The default value is 10.
-
-tune.quic.reorder-ratio <0..100, in percent>
- The ratio applied to the packet reordering threshold calculated. It may
- trigger a high packet loss detection when too small.
-
- The default value is 50.
-
tune.quic.retry-threshold <number>
Dynamically enables the Retry feature for all the configured QUIC listeners
as soon as this number of half open connections is reached. A half open
size_t quic_frontend_max_window_size;
unsigned int quic_frontend_stream_data_ratio;
unsigned int quic_retry_threshold;
- unsigned int quic_reorder_ratio;
- unsigned int quic_max_frame_loss;
- unsigned int quic_cubic_loss_tol;
#endif /* USE_QUIC */
} tune;
struct {
#define QUIC_RETRY_DURATION_SEC 10
/* Default Retry threshold */
#define QUIC_DFLT_RETRY_THRESHOLD 100 /* in connection openings */
-/* Default ratio value applied to a dynamic Packet reorder threshold. */
-#define QUIC_DFLT_REORDER_RATIO 50 /* in percent */
-/* Default limit of loss detection on a single frame. If exceeded, connection is closed. */
-#define QUIC_DFLT_MAX_FRAME_LOSS 10
/* Default congestion window size. 480 kB, equivalent to the legacy value which was 30*bufsize */
#define QUIC_DFLT_MAX_WINDOW_SIZE 491520
#error "Must define USE_OPENSSL"
#endif
+
+/* Default limit of loss detection on a single frame. If exceeded, connection is closed. */
+#define QUIC_DFLT_CC_MAX_FRAME_LOSS 10
+/* Default ratio value applied to a dynamic Packet reorder threshold. */
+#define QUIC_DFLT_CC_REORDER_RATIO 50 /* in percent */
+
+
#define QUIC_TUNE_FE_LISTEN_OFF 0x00000001
#define QUIC_TUNE_NO_PACING 0x00000001
#define QUIC_TUNE_NO_UDP_GSO 0x00000002
#define QUIC_TUNE_SOCK_PER_CONN 0x00000004
-#define QUIC_TUNE_CC_HYSTART 0x00000008
+
+#define QUIC_TUNE_FB_CC_HYSTART 0x00000001
struct quic_tune {
struct {
+ uint cc_cubic_min_losses;
+ uint cc_max_frame_loss;
+ uint cc_reorder_ratio;
uint sec_glitches_threshold;
uint opts; /* QUIC_TUNE_FE_* options specific to FE side */
uint fb_opts; /* QUIC_TUNE_FB_* options shared by both side */
} fe;
struct {
+ uint cc_cubic_min_losses;
+ uint cc_max_frame_loss;
+ uint cc_reorder_ratio;
uint sec_glitches_threshold;
uint fb_opts; /* QUIC_TUNE_FB_* options shared by both side */
} be;
#define QUIC_CC_BBR_STR "bbr"
#define QUIC_CC_NO_CC_STR "nocc"
-struct quic_tune quic_tune;
+struct quic_tune quic_tune = {
+ .fe = {
+ .cc_max_frame_loss = QUIC_DFLT_CC_MAX_FRAME_LOSS,
+ .cc_reorder_ratio = QUIC_DFLT_CC_REORDER_RATIO,
+ },
+ .be = {
+ .cc_max_frame_loss = QUIC_DFLT_CC_MAX_FRAME_LOSS,
+ .cc_reorder_ratio = QUIC_DFLT_CC_REORDER_RATIO,
+ },
+};
static int bind_parse_quic_force_retry(char **args, int cur_arg, struct proxy *px, struct bind_conf *conf, char **err)
{
}
suffix = args[0] + prefix_len;
- if (strcmp(suffix, "cc.cubic.min-losses") == 0)
- global.tune.quic_cubic_loss_tol = arg - 1;
+ if (strcmp(suffix, "be.cc.cubic-min-losses") == 0 ||
+ strcmp(suffix, "fe.cc.cubic-min-losses") == 0) {
+ uint *ptr = (suffix[0] == 'b') ? &quic_tune.be.cc_cubic_min_losses :
+ &quic_tune.fe.cc_cubic_min_losses;
+ *ptr = arg - 1;
+ }
+ else if (strcmp(suffix, "be.cc.max-frame-loss") == 0 ||
+ strcmp(suffix, "fe.cc.max-frame-loss") == 0) {
+ uint *ptr = (suffix[0] == 'b') ? &quic_tune.be.cc_max_frame_loss :
+ &quic_tune.fe.cc_max_frame_loss;
+ *ptr = arg;
+ }
+ else if (strcmp(suffix, "be.cc.reorder-ratio") == 0 ||
+ strcmp(suffix, "fe.cc.reorder-ratio") == 0) {
+ uint *ptr = (suffix[0] == 'b') ? &quic_tune.be.cc_reorder_ratio :
+ &quic_tune.fe.cc_reorder_ratio;
+ *ptr = arg;
+ }
else if (strcmp(suffix, "be.sec.glitches-threshold") == 0 ||
strcmp(suffix, "fe.sec.glitches-threshold") == 0) {
uint *ptr = (suffix[0] == 'b') ? &quic_tune.be.sec_glitches_threshold :
}
global.tune.quic_frontend_stream_data_ratio = arg;
}
- else if (strcmp(suffix, "max-frame-loss") == 0)
- global.tune.quic_max_frame_loss = arg;
- else if (strcmp(suffix, "reorder-ratio") == 0) {
- if (arg > 100) {
- memprintf(err, "'%s' expects an integer argument between 0 and 100.", args[0]);
- return -1;
- }
-
- global.tune.quic_reorder_ratio = arg;
- }
else if (strcmp(suffix, "retry-threshold") == 0)
global.tune.quic_retry_threshold = arg;
/* legacy options */
+ else if (strcmp(suffix, "cc.cubic.min-losses") == 0) {
+ memprintf(err, "'%s' is deprecated in 3.3 and will be removed in 3.5. "
+ "Please use the newer keyword syntax 'tune.quic.fe.cc.cubic-min-losses'.", args[0]);
+ quic_tune.fe.cc_cubic_min_losses = arg - 1;
+ ret = 1;
+ }
else if (strcmp(suffix, "frontend.glitches-threshold") == 0) {
memprintf(err, "'%s' is deprecated in 3.3 and will be removed in 3.5. "
"Please use the newer keyword syntax 'tune.quic.fe.sec.glitches-threshold'.", args[0]);
quic_tune.fe.sec_glitches_threshold = arg;
ret = 1;
}
+ else if (strcmp(suffix, "max-frame-loss") == 0) {
+ memprintf(err, "'%s' is deprecated in 3.3 and will be removed in 3.5. "
+ "Please use the newer keyword syntax 'tune.quic.fe.cc.max-frame-loss'.", args[0]);
+ quic_tune.fe.cc_max_frame_loss = arg;
+ ret = 1;
+ }
+ else if (strcmp(suffix, "reorder-ratio") == 0) {
+ memprintf(err, "'%s' is deprecated in 3.3 and will be removed in 3.5. "
+ "Please use the newer keyword syntax 'tune.quic.fe.cc.reorder-ratio'.", args[0]);
+ if (arg > 100) {
+ memprintf(err, "'%s' expects an integer argument between 0 and 100.", args[0]);
+ return -1;
+ }
+
+ quic_tune.fe.cc_reorder_ratio = arg;
+ ret = 1;
+ }
else {
memprintf(err, "'%s' keyword not unhandled (please report this bug).", args[0]);
return -1;
int on;
int prefix_len = strlen("tune.quic.");
const char *suffix;
+ int ret = 0;
if (too_many_args(1, args, err, NULL))
return -1;
else
global.tune.no_zero_copy_fwd |= NO_ZERO_COPY_FWD_QUIC_SND;
}
+ else if (strcmp(suffix, "be.cc.hystart") == 0 ||
+ strcmp(suffix, "fe.cc.hystart") == 0) {
+ uint *ptr = (suffix[0] == 'b') ? &quic_tune.be.fb_opts :
+ &quic_tune.fe.fb_opts;
+ if (on)
+ *ptr |= QUIC_TUNE_FB_CC_HYSTART;
+ else
+ *ptr &= ~QUIC_TUNE_FB_CC_HYSTART;
+ }
else if (strcmp(suffix, "cc-hystart") == 0) {
+ memprintf(err, "'%s' is deprecated in 3.3 and will be removed in 3.5. "
+ "Please use the newer keyword syntax 'tune.quic.fe.cc.hystart'.", args[0]);
if (on)
- quic_tune.options |= QUIC_TUNE_CC_HYSTART;
+ quic_tune.fe.fb_opts |= QUIC_TUNE_FB_CC_HYSTART;
else
- quic_tune.options &= ~QUIC_TUNE_CC_HYSTART;
+ quic_tune.fe.fb_opts &= ~QUIC_TUNE_FB_CC_HYSTART;
+ ret = 1;
}
- return 0;
+ return ret;
}
static struct cfg_kw_list cfg_kws = {ILH, {
{ CFG_GLOBAL, "tune.quic.listen", cfg_parse_quic_tune_on_off },
{ CFG_GLOBAL, "tune.quic.socket-owner", cfg_parse_quic_tune_socket_owner },
- { CFG_GLOBAL, "tune.quic.cc-hystart", cfg_parse_quic_tune_on_off },
- { CFG_GLOBAL, "tune.quic.cc.cubic.min-losses", cfg_parse_quic_tune_setting },
{ CFG_GLOBAL, "tune.quic.frontend.max-data-size", cfg_parse_quic_tune_setting },
{ CFG_GLOBAL, "tune.quic.frontend.max-streams-bidi", cfg_parse_quic_tune_setting },
{ CFG_GLOBAL, "tune.quic.frontend.max-idle-timeout", cfg_parse_quic_time },
{ CFG_GLOBAL, "tune.quic.frontend.max-tx-mem", cfg_parse_quic_tune_setting },
{ CFG_GLOBAL, "tune.quic.frontend.default-max-window-size", cfg_parse_quic_tune_setting },
{ CFG_GLOBAL, "tune.quic.frontend.stream-data-ratio", cfg_parse_quic_tune_setting },
- { CFG_GLOBAL, "tune.quic.max-frame-loss", cfg_parse_quic_tune_setting },
- { CFG_GLOBAL, "tune.quic.reorder-ratio", cfg_parse_quic_tune_setting },
{ CFG_GLOBAL, "tune.quic.retry-threshold", cfg_parse_quic_tune_setting },
{ CFG_GLOBAL, "tune.quic.disable-tx-pacing", cfg_parse_quic_tune_setting0 },
{ CFG_GLOBAL, "tune.quic.disable-udp-gso", cfg_parse_quic_tune_setting0 },
{ CFG_GLOBAL, "tune.quic.zero-copy-fwd-send", cfg_parse_quic_tune_on_off },
+ { CFG_GLOBAL, "tune.quic.fe.cc.cubic-min-losses", cfg_parse_quic_tune_setting },
+ { CFG_GLOBAL, "tune.quic.fe.cc.hystart", cfg_parse_quic_tune_on_off },
+ { CFG_GLOBAL, "tune.quic.fe.cc.max-frame-loss", cfg_parse_quic_tune_setting },
+ { CFG_GLOBAL, "tune.quic.fe.cc.reorder-ratio", cfg_parse_quic_tune_setting },
{ CFG_GLOBAL, "tune.quic.fe.sec.glitches-threshold", cfg_parse_quic_tune_setting },
+ { CFG_GLOBAL, "tune.quic.be.cc.cubic-min-losses", cfg_parse_quic_tune_setting },
+ { CFG_GLOBAL, "tune.quic.be.cc.hystart", cfg_parse_quic_tune_on_off },
+ { CFG_GLOBAL, "tune.quic.be.cc.max-frame-loss", cfg_parse_quic_tune_setting },
+ { CFG_GLOBAL, "tune.quic.be.cc.reorder-ratio", cfg_parse_quic_tune_setting },
{ CFG_GLOBAL, "tune.quic.be.sec.glitches-threshold", cfg_parse_quic_tune_setting },
/* legacy options */
+ { CFG_GLOBAL, "tune.quic.cc-hystart", cfg_parse_quic_tune_on_off },
+ { CFG_GLOBAL, "tune.quic.cc.cubic.min-losses", cfg_parse_quic_tune_setting },
{ CFG_GLOBAL, "tune.quic.frontend.glitches-threshold", cfg_parse_quic_tune_setting },
+ { CFG_GLOBAL, "tune.quic.max-frame-loss", cfg_parse_quic_tune_setting },
+ { CFG_GLOBAL, "tune.quic.reorder-ratio", cfg_parse_quic_tune_setting },
{ 0, NULL, NULL }
}};
.quic_frontend_max_tx_mem = QUIC_MAX_TX_MEM,
.quic_frontend_max_window_size = QUIC_DFLT_MAX_WINDOW_SIZE,
.quic_frontend_stream_data_ratio = QUIC_DFLT_FRONT_STREAM_DATA_RATIO,
- .quic_reorder_ratio = QUIC_DFLT_REORDER_RATIO,
.quic_retry_threshold = QUIC_DFLT_RETRY_THRESHOLD,
- .quic_max_frame_loss = QUIC_DFLT_MAX_FRAME_LOSS,
#endif /* USE_QUIC */
},
#ifdef USE_OPENSSL
c->last_w_max = 0;
c->W_est = 0;
c->recovery_start_time = 0;
- if (quic_tune.options & QUIC_TUNE_CC_HYSTART)
+ if (quic_tune_test(QUIC_TUNE_FB_CC_HYSTART, cc->qc))
quic_cc_hystart_reset(&c->hystart);
TRACE_LEAVE(QUIC_EV_CONN_CC, cc->qc);
}
TRACE_PROTO("CC cubic", QUIC_EV_CONN_CC, cc->qc, ev);
switch (ev->type) {
case QUIC_CC_EVT_ACK:
- if (quic_tune.options & QUIC_TUNE_CC_HYSTART) {
+ if (quic_tune_test(QUIC_TUNE_FB_CC_HYSTART, cc->qc)) {
struct quic_hystart *h = &c->hystart;
unsigned int acked = QUIC_MIN(ev->ack.acked, (uint64_t)HYSTART_LIMIT * path->mtu);
* though.
*/
c->consecutive_losses += ev->loss.count;
- if (c->consecutive_losses <= global.tune.quic_cubic_loss_tol)
+ if (c->consecutive_losses <= QUIC_TUNE_FB_GET(cc_cubic_min_losses, cc->qc))
goto out;
quic_enter_recovery(cc);
break;
#include <haproxy/quic_loss.h>
#include <haproxy/quic_tls.h>
#include <haproxy/quic_trace.h>
+#include <haproxy/quic_tune.h>
#include <haproxy/atomic.h>
#include <haproxy/list.h>
struct quic_loss *ql;
unsigned int loss_delay;
uint64_t pktthresh;
+ int reorder_ratio;
TRACE_ENTER(QUIC_EV_CONN_PKTLOSS, qc);
TRACE_PROTO("TX loss", QUIC_EV_CONN_PKTLOSS, qc, pktns);
* flight before loss detection.
*/
pktthresh = pktns->tx.next_pn - 1 - eb64_entry(node, struct quic_tx_packet, pn_node)->pn_node.key;
+ reorder_ratio = QUIC_TUNE_FB_GET(cc_reorder_ratio, qc);
/* Apply a ratio to this threshold and add it to QUIC_LOSS_PACKET_THRESHOLD. */
- pktthresh = pktthresh * global.tune.quic_reorder_ratio / 100 + QUIC_LOSS_PACKET_THRESHOLD;
+ pktthresh = pktthresh * reorder_ratio / 100 + QUIC_LOSS_PACKET_THRESHOLD;
while (node) {
struct quic_tx_packet *pkt;
int64_t largest_acked_pn;
#include <haproxy/quic_tls.h>
#include <haproxy/quic_token.h>
#include <haproxy/quic_trace.h>
+#include <haproxy/quic_tune.h>
#include <haproxy/quic_tx.h>
#include <haproxy/quic_utils.h>
#include <haproxy/ssl_sock.h>
qc_frm_free(qc, &frm);
}
else {
- if (++frm->loss_count >= global.tune.quic_max_frame_loss) {
+ const uint max_frame_loss = QUIC_TUNE_FB_GET(cc_max_frame_loss, qc);
+
+ if (++frm->loss_count >= max_frame_loss) {
TRACE_ERROR("retransmission limit reached, closing the connection", QUIC_EV_CONN_PRSAFRM, qc);
quic_set_connection_close(qc, quic_err_transport(QC_ERR_INTERNAL_ERROR));
qc_notify_err(qc);
}
qc->path->in_flight += pkt->in_flight_len;
pkt->pktns->tx.in_flight += pkt->in_flight_len;
- if ((quic_tune.options & QUIC_TUNE_CC_HYSTART) && pkt->pktns == qc->apktns)
+ if (quic_tune_test(QUIC_TUNE_FB_CC_HYSTART, qc) && pkt->pktns == qc->apktns)
cc->algo->hystart_start_round(cc, pkt->pn_node.key);
if (pkt->in_flight_len)
qc_set_timer(qc);