From: Willy Tarreau <w@1wt.eu>
Date: Wed, 25 Jul 2018 13:21:00 +0000 (+0200)
Subject: DOC: queue: document the expected locking model for the server's queue
X-Git-Tag: v1.9-dev1~24
X-Git-Url: http://git.ipfire.org/?a=commitdiff_plain;h=6bdd05c0ef6d7c3208f1a9f6d1cb7b6edc2b90ad;p=thirdparty%2Fhaproxy.git

DOC: queue: document the expected locking model for the server's queue

The locking model is not trivial and is worth documenting to avoid seeing
apparent bugs everywhere while they are not.
---

diff --git a/src/queue.c b/src/queue.c
index c4f7366412..d522f0e91a 100644
--- a/src/queue.c
+++ b/src/queue.c
@@ -10,6 +10,72 @@
  *
  */
 
+/* Short explanation on the locking, which is far from being trivial : a
+ * pendconn is a list element which necessarily is associated with an existing
+ * stream. It has pendconn->strm always valid. A pendconn may only be in one of
+ * these three states :
+ *   - unlinked : in this case it is an empty list head ;
+ *   - linked into the server's queue ;
+ *   - linked into the proxy's queue.
+ *
+ * A stream does not necessarily have such a pendconn. Thus the pendconn is
+ * designated by the stream->pend_pos pointer. This results in some properties :
+ *   - pendconn->strm->pend_pos is never NULL for any valid pendconn
+ *   - if LIST_ISEMPTY(pendconn->list) is true, the element is unlinked,
+ *     otherwise it necessarily belongs to one of the other lists ; this may
+ *     not be atomically checked under threads though ;
+ *   - pendconn->px is never NULL if pendconn->list is not empty
+ *   - pendconn->sv is never NULL if pendconn->list is in the server's queue,
+ *     and is always NULL if pendconn->list is in the backend's queue or empty.
+ *
+ * Threads complicate the design a little bit but rules remain simple :
+ *   - a lock is present in the pendconn (pendconn->lock). This lock is
+ *     used to protect pendconn->list when accessing from the pendconn,
+ *     and to ensure srv/px are not changing at the same time. This lock
+ *     must be held while adding/removing the pendconn to/from a queue.
+ *
+ *   - the server's queue lock must be held at least when manipulating the
+ *     server's queue, which is when adding a pendconn to the queue and when
+ *     removing a pendconn from the queue. It protects the queue's integrity.
+ *
+ *   - the proxy's queue lock must be held at least when manipulating the
+ *     proxy's queue, which is when adding a pendconn to the queue and when
+ *     removing a pendconn from the queue. It protects the queue's integrity.
+ *
+ *   - all three locks are compatible and may be held at the same time.
+ *
+ *   - a pendconn_add() is only performed by the stream which will own the
+ *     pendconn ; the pendconn is allocated at this moment and returned ; it is
+ *     added to either the server or the proxy's queue while holding this
+ *     queue's lock. The pendconn lock is never taken here since the pendconn
+ *     is new and cannot be met by another thread before being inserted.
+ *
+ *   - the pendconn is then met by a thread walking over the proxy or server's
+ *     queue with the respective lock held. This lock is exclusive and the
+ *     pendconn can only appear in one queue so by definition a single thread
+ *     may find this pendconn at a time.
+ *
+ *   - the pendconn is unlinked either by its own stream upon success/abort/
+ *     free, or by another one offering it its server slot. This is achieved by
+ *     pendconn_process_next_strm() under either the server or proxy's lock,
+ *     pendconn_redistribute() under the server's lock, pendconn_grab_from_px()
+ *     under the proxy's lock, or pendconn_unlink() under either the proxy's or
+ *     the server's lock depending on the queue the pendconn is attached to.
+ *
+ *   - no single operation except the pendconn initialisation prior to the
+ *     insertion are performed without a queue lock held.
+ *
+ *   - pendconn_grab_from_px() and pendconn_process_next_strm() assign ->srv so
+ *     that the stream knows what server to work with (via pendconn_dequeue()).
+ *     It's worth noting that this assignment complicates the detection of the
+ *     queue to be locked/unlocked.
+ *
+ *   - a pendconn doesn't switch between queues, it stays where it is.
+ *
+ *   - strm->pend_pos is assigned late so pendconn->strm->pend_pos could be met
+ *     uninitialized by another thread and must not be relied on.
+ */
+
 #include <common/config.h>
 #include <common/memory.h>
 #include <common/time.h>