BEE Backport bacula/src/stored/stored_conf.c

BEE Backport bacula/src/stored/stored_conf.c

This commit is the result of the squash of the following main commits:

Author: Eric Bollengier <eric@baculasystems.com>
Date:   Wed Mar 4 15:41:15 2020 +0100

    Fix compilation of community version + move dedup configuration functions to external file

Author: Eric Bollengier <eric@baculasystems.com>
Date:   Wed Feb 5 12:00:41 2020 +0100

    Fix *json segfault on OpenBSD

    In memcpy(&res_all, res, sizeof(res_all));

    The res_all is likely bigger than res itself. We
    copy too much data, and on OpenBSD, we get segfaults.

Author: Alain Spineux <alain@baculasystems.com>
Date:   Wed Nov 20 13:32:52 2019 +0100

    dedup: detect and report dedupengine startup failure

    - 2 cases to handle
      - cannot load the driver
      - error at the dedupengine creation time or startup
    - the error message is stored in the dedup ressourse
    - see regress/tests/dedup-start-fail-test for more

    # Conflicts:
    #       bacula/src/stored/dircmd.c

Author: Eric Bollengier <eric@baculasystems.com>
Date:   Fri Nov 15 09:12:29 2019 +0100

    cloud: Rename TransferRetentionDays to TransferRetention (expressed as a time)

Author: Alain Spineux <alain@baculasystems.com>
Date:   Tue Oct 29 16:16:14 2019 +0100

    dedup: rename Dedup resource (and directive) into Dedupengine

Author: Alain Spineux <alain@baculasystems.com>
Date:   Thu Oct 24 16:23:10 2019 +0200

    dedup: create a Dedup resource based on the dedup directive in Storage

    - to stay compatible with old dedup directive in the Storage resource

Author: Norbert Bizet <norbert.bizet@baculasystems.com>
Date:   Mon Oct 14 05:35:57 2019 -0400

    cloud: handle restore params in Cloud res

Author: Alain Spineux <alain@baculasystems.com>
Date:   Tue Sep 24 10:17:51 2019 +0200

    dedup: move all dedup code into the plugin

    - remove global variables: dedupengine and bucker_manager
    - update protocol SD<->DIR
    - extend dedup_dev and BufferedMsgSD to suuport all function
    - create "dummy" functions in dev.h
    Warning :
    - include a dedupengine use counter
    - .status storage=XXX dedupengine display all the dedupengine on the SD
     instead of the one(s) related to the "device"

Author: Alain Spineux <alain@baculasystems.com>
Date:   Thu Jun 6 11:59:15 2019 +0200

    Fix #3987 Scrub limit set in the storage{} not taken into account

    - we were using two different variables

Author: Alain Spineux <alain@baculasystems.com>
Date:   Tue Apr 9 17:43:43 2019 +0200

    PSK: set default "TLS PSK Enable" to "no" when SSL is not available

Author: Alain Spineux <alain@baculasystems.com>
Date:   Mon Apr 8 14:56:33 2019 +0200

    PSK: Add new "TLS PSK Enable" directive to all resources

    - add the field to the resources
    - create default "psk_ctx" CONTEXT for each of them at startup

Author: Henrique <henrique.faria@baculasystems.com>
Date:   Sun Jan 6 20:45:39 2019 -0200

    cloud: added Swift Cloud Plugin

Author: Eric Bollengier <eric@baculasystems.com>
Date:   Thu Nov 1 17:33:35 2018 +0100

    cloud: Accept empty Hostname, SecretKey and AccessKey for Generic drivers

Author: Eric Bollengier <eric@baculasystems.com>
Date:   Thu Sep 6 17:14:02 2018 +0200

    Add SSDDirectory resource to the Storage Daemon

Author: Norbert Bizet <norbert.bizet@baculasystems.com>
Date:   Thu Jul 26 10:27:31 2018 +0200

    cloud: Add google and oracle cloud directives handling

    driver files should be located in plugins_directory.

Author: Norbert Bizet <norbert.bizet@baculasystems.com>
Date:   Wed Jun 27 18:27:23 2018 +0200

    cloud: Allow Generic driver in configration

Author: Norbert Bizet <norbert.bizet@baculasystems.com>
Date:   Thu Jun 21 18:19:05 2018 +0200

    cloud: First implementation and scenario test for generic_driver

Author: Radosław Korzeniewski <radekk@inteos.pl>
Date:   Fri Jun 1 09:52:18 2018 +0200

    Add a Collector resource and threads to SD.

Author: Marcin Haba <marcin.haba@bacula.pl>
Date:   Mon Nov 6 14:31:44 2017 +0100

    Remove default value flag from the cloud endpoint directives

Author: Norbert Bizet <norbert.bizet@baculasystems.com>
Date:   Mon Sep 11 15:45:15 2017 +0200

    cloud: Add callbacks to was_driver

    This allows to cancel processed when possible and report progress.
    cloud driver init() interface is modified to pass usefull block size info.

Author: Norbert Bizet <norbert.bizet@baculasystems.com>
Date:   Mon Aug 28 17:01:53 2017 +0200

    Azure : correct endpoint handling in conf files.

Author: Norbert Bizet <norbert.bizet@baculasystems.com>
Date:   Wed Aug 9 16:07:47 2017 +0200

    azure: was_driver class and C_WAS_DRIVER enum.

Author: Norbert Bizet <norbert.bizet@baculasystems.com>
Date:   Wed Aug 9 15:52:13 2017 +0200

    azure: add some azure-specific resoures to CLOUD

Author: Norbert Bizet <norbert.bizet@baculasystems.com>
Date:   Mon May 22 16:36:40 2017 +0200

    cloud: Add "Manual" cloud upload option and add truncate option to the upload command.

Author: Eric Bollengier <eric@baculasystems.com>
Date:   Mon May 1 09:39:57 2017 +0200

    dedup: Add directive Storage::DedupCheckHash to control the rehydration control. Default is no.

Author: Kern Sibbald <kern@sibbald.com>
Date:   Mon Oct 31 18:17:19 2016 +0100

    Implement Cloud resource TruncateCache and Synchronize directives (not used yet)

Author: Alain Spineux <alain@baculasystems.com>
Date:   Wed Sep 28 16:44:27 2016 +0200

    dedup: add MaxContainerSize directive to SD ( FIX #1728 )

    - this directive must limit the size of the container
    - add max_container_size to bacula-sd.conf and to BucketManager
    - new method BucketManager::grow0() like grow() but without any lock
    - new method BucketManager::add_buckets() that add a bucket
    - new method BucketManager::can_grow_more() that tell if a bucket can grow
    - allocate the max size (511 entries) for the container table
      (this is the maximum for standard 64K header)
    - use BucketManager::alloc_block() instead of Bucket::alloc_block() to alloc chunks
      BucketManager::alloc_block() will handle the choice of the bucket, extend or
      create a new one when required.
    - use Bucket::capacity() instead of ba.size
    - dedup: link extra container together and keep ba.size < max_container_bidx
      . container of the same max_blocksize are linked
      . the head is got by BucketManager::choose_bucket()
      . the current container is head->current
      . they can be iterated up to NULL vi head->current->next
      . ba.size is kept < max_container_bidx by grow0()
        (I started doing a "runtime" limitation in alloc_block())
      . brc32 must use the right size (ba.size,max_container_bidx) at startup
      . add_buckets() return a bucket and handle "md_fsmonly"
    - extend clone_fsm() to handle extra containers
    - extend BucketManager::alloc_block() to keep vacuum_fsm geometry in sync
    - add a lock parameter to BucketManager::get_bucket(blockaddr addr, bool lck=true)
    - new method DedupEngine::vacuum_mark0() with a lock parameter
    - can modifying MaxContainerSize at any time
     . containers that are already bigger than the new value will no grow anymore
     . container that are smaller than the new value will continue to grow
    - add bucketmanager::Check

Author: Kern Sibbald <kern@sibbald.com>
Date:   Thu Sep 1 20:16:25 2016 +0200

    First cut Cloud resource for SD Device

Author: Kern Sibbald <kern@sibbald.com>
Date:   Thu Jul 28 09:57:32 2016 +0200

    Get device name tables setup correctly + some comments

Author: Eric Bollengier <eric@baculasystems.com>
Date:   Mon Jul 18 17:40:51 2016 +0200

    Change the default MaximumFileIndex from 50MB to 100MB by default

Author: Kern Sibbald <kern@sibbald.com>
Date:   Thu Jun 16 18:40:26 2016 +0200

    Implement driver classes + loadable drivers + first cut cloud driver

Author: Alain Spineux <alain@baculasystems.com>
Date:   Tue Jun 3 15:51:12 2014 +0200

    dedup: handle purge part1

    just rename the "volume" into "volumeXXX.tobepurged"

Author: Kern Sibbald <kern@sibbald.com>
Date:   Thu May 8 10:09:13 2014 +0200

    Redefine SD dedup directives

Author: Alain Spineux <alain@baculasystems.com>
Date:   Sun Apr 6 20:05:35 2014 +0200

    Beginning of dedup code

Author: Kern Sibbald <kern@sibbald.com>
Date:   Sat Dec 10 17:53:22 2011 +0100

    Cleanup compiler warnings