[collectd 6] src/daemon/plugin.c: Use one thread per write plugin

[collectd 6] src/daemon/plugin.c: Use one thread per write plugin

ChangeLog: collectd: Use one write thread per write plugin

The previous write thread design used a single queue with a single read head
from which one of the write threads would de-queue an element and would then
sequentially call each registered write callback.
This meant that all write plugins would have to cooperate in order to not drop
values. If for example all write threads are stalled by the same write plugin's
callback function not returning, the queue will start to fill up until elements
start to be dropped, even though there are other plugins that could still make
progress. In addition to that, all write callbacks have to be designed to be
reentrant right now, which increases complexity.

This new design uses a single linked-list write queue with one read head per
output plugin. Each output plugin is serviced in a dedicated write thread.
Elements are freed based on a reference count, which is shown in the ASCII-Art
below:

          +- Thread #1 Head       +- Thread #2 Head       +- Tail
          v                       v                       v
   +--+  +--+  +--+  +--+  +--+  +--+  +--+  +--+  +--+  +--+
   | 0|->| 1|->| 1|->| 1|->| 1|->| 2|->| 2|->| 2|->| 2|->| 2|->X
   +--+  +--+  +--+  +--+  +--+  +--+  +--+  +--+  +--+  +--+
    ^
    +- to be free()d

The changes introduced by this commit have some side-effects:

- The WriteThreads config option no longer exists, as a strict 1:1 ratio of
  write plugins and write threads is used.
- The data flow has changed. The previous data flow was:

    (From one of the ReadThreads)
      plugin_dispatch_{values,multivalue}()
      plugin_dispatch_metric_family()
      enqueue_metric_family()
      write_queue_enqueue() -----{Queue}----+
                                            |
    (In one of the WriteThreads threads)    |
      plugin_write_thread()                 |
        ^- plugin_write_dequeue()         <-+
      plugin_dispatch_metric_internal()
        ^- fc_process_chain(pre_cache_chain)
      fc_process_chain(fc_process_chain)
      fc_bit_write_invoke()
      plugin_write(NULL) / plugin_write(plugin_name)
      plugin callback()

  The data flow now is:

    (From one of the ReadThreads)
      plugin_dispatch_{values,multivalue}()
      plugin_dispatch_metric_family()
      plugin_dispatch_metric_internal()
        ^- fc_process_chain(pre_cache_chain)
      fc_process_chain(post_cache_chain)
      fc_bit_write_invoke()
      plugin_write(NULL) / plugin_write(plugin_name)
      write_queue_enqueue() -----{Queue}----+
                                            |
    (In one of the WriteThreads threads)    |
      plugin_write_thread()               <-+
      plugin callback()

  One result of this change is, that the behaviour of plugin_write has changed
  from running the plugin callback immediately and in the same thread, to
  always enqueueing the value and de-queing in the dedicated thread.
- The behaviour of the WriteQueueLimitHigh and WriteQueueLimitLow options has
  changed. The Queue will be be capped to a length of LimitHigh by dropping
  random queue elements between the queue end and LimitLow.
  Setting LimitLow to a reasonably large value ensures that fast write plugins
  do not loose values, even in the vicinity of a slow plugin.
  The diagram below shows the random element selected for removal (###) in
  Step 1 and the queue with the element removed in Step 2.

  Step 1:
    +- Thread #1 Head    |  +- Thread #2 Head       +- Tail
    v  |                 |  v                       v
   +--+| +--+  ####  +--+| +--+  +--+  +--+  +--+  +--+
   | 1|->| 1|-># 1#->| 1|->| 2|->| 2|->| 2|->| 2|->| 2|->X
   +--+| +--+  ####  +--+| +--+  +--+  +--+  +--+  +--+
       |                 |
       | LimitHigh       | LimitLow

  Step 2:
       |  +- Thread #1 Head +- Thread #2 Head       +- Tail
       |  v              |  v                       v
       | +--+  +--+  +--+| +--+  +--+  +--+  +--+  +--+
       | | 1|->| 1|->| 1|->| 2|->| 2|->| 2|->| 2|->| 2|->X
       | +--+  +--+  +--+| +--+  +--+  +--+  +--+  +--+
       |                 |
       | LimitHigh       | LimitLow

Signed-off-by: Leonard Göhrs <l.goehrs@pengutronix.de>