Stephan Bosch [Tue, 2 Jul 2019 18:50:07 +0000 (20:50 +0200)]
lib: event-log - Add support for inserting a prefix in event_logv().
The prefix is provided in the event_logv() call and not attached to the event
hierarchy. It is inserted at the position indicated by the base_event.
Different prefixes can be provided for the message appended to the string buffer
and the sent event message.
Stephan Bosch [Sun, 30 Jun 2019 09:45:16 +0000 (11:45 +0200)]
lib: event-log - Add support for appending part of the log message to a string buffer while sending the event.
The message appended to the string buffer includes prefixes and message callback
modifications by parent events up until a specified base event. The event is
otherwise sent as normal with the full prefixes and all modifications up to the
root event, so in that respect this new feature behaves just like event_log*().
This is primarily useful to mimic (part of) event logging in parallel logs that
are visible to users. It is also possible to inhibit sending the event and only
log to the string buffer.
Stephan Bosch [Mon, 1 Jul 2019 23:04:03 +0000 (19:04 -0400)]
lib-http: guard against hshared use-after-free
This fixes a race condition where the http_client_host_shared_idle_timeout()
function would get called with an already freed hshared argument.
Specifically, the situation arises from the hshared idle timeout calling
http_client_host_shared_free(), which removes the timeout and then proceeds to
free the client queue. The client queue freeing code indirectly calls
http_client_host_shared_check_idle(), which notices that there is no idle
timeout and allocates one.
The backtrace at the point of this new timeout allocation:
frame #3: 0x00007f0c775897f0 libdovecot.so.0`timeout_add_to(...) ioloop.c:280
frame #4: 0x00007f0c7751a45f libdovecot.so.0`http_client_host_shared_check_idle(hshared=<unavailable>) at http-client-host.c:69
frame #5: 0x00007f0c7750de89 libdovecot.so.0`http_client_request_error(_req=<unavailable>, status=9000, error="") at http-client-request.c:1525
frame #6: 0x00007f0c77517f38 libdovecot.so.0`http_client_queue_fail_full(queue=0x000055e13cff0e10, status=9000, error="", all=<unavailable>) at http-client-queue.c:183
frame #7: 0x00007f0c77518baa libdovecot.so.0`http_client_queue_free(queue=0x000055e13cff0e10) at http-client-queue.c:141
frame #8: 0x00007f0c7751a8bc libdovecot.so.0`http_client_host_free_shared(_host=<unavailable>) at http-client-host.c:391
frame #9: 0x00007f0c7751ab4c libdovecot.so.0`http_client_host_shared_free(_hshared=0x00007ffdac109e48) at http-client-host.c:294
frame #10: 0x00007f0c7751ace8 libdovecot.so.0`http_client_host_shared_idle_timeout(hshared=<unavailable>) at http-client-host.c:40
frame #11: 0x00007f0c7758a1a4 libdovecot.so.0`io_loop_handle_timeouts at ioloop.c:682
frame #12: 0x00007f0c7758a089 libdovecot.so.0`io_loop_handle_timeouts(ioloop=0x000055e13cfc8d80) at ioloop.c:696
frame #13: 0x00007f0c7758befc libdovecot.so.0`io_loop_handler_run_internal(ioloop=0x000055e13cfc8d80) at ioloop-select.c:126
frame #14: 0x00007f0c7758a56d libdovecot.so.0`io_loop_handler_run(ioloop=<unavailable>) at ioloop.c:767
frame #15: 0x00007f0c7758a798 libdovecot.so.0`io_loop_run(ioloop=0x000055e13cfc8d80) at ioloop.c:740
frame #16: 0x00007f0c774f61eb libdovecot.so.0`master_service_run(service=0x000055e13cfc8c10, callback=<unavailable>) at master-service.c:782
frame #17: 0x000055e13b48e3a5 stats`main(argc=<unavailable>, argv=<unavailable>) at main.c:99
frame #18: 0x00007f0c771092e1 libc.so.6`__libc_start_main + 241
frame #19: 0x000055e13b48e41a stats`_start + 42
Timo Sirainen [Mon, 8 Jul 2019 15:59:21 +0000 (18:59 +0300)]
lib-index: Fix using old map when checking for unexpected changes during log rotate
This shouldn't normally be possible. The log is locked, so other processes
shouldn't be able to write anything to it. This was mainly found by the
earlier bug.
Timo Sirainen [Mon, 8 Jul 2019 15:56:12 +0000 (18:56 +0300)]
lib-index: After recreating cache, make sure offsets are immediately updated to map
They were most likely refreshed anyway before the next cache usage, but this
caused an assert-crash if the dovecot.index was also recreated in the same
sync.
Add the ability to prevent dovecot/lmtp from adding received header. This is
useful if one needs to use LMTP, but does not want additional received headers
because:
1) Dovecot does not have correct information to add to the headers, and causes
misleading/incorrect information
2) Does not want to expose internal information
The default value is true, which is equal to the current behavior.
Shaun A. Johnson [Mon, 13 Aug 2018 23:43:04 +0000 (16:43 -0700)]
lib: Changed chown to chmod for Linux NFS Flush
This change switches the chown call to a chmod call in the function
nfs_flush_chown_uid because the chown call was resulting in certain
permission bits being dropped when the nfs attributes were flushed.
Functionality for FreeBSD/Solaris was left unchanged.
Shaun A. Johnson [Mon, 13 Aug 2018 23:35:54 +0000 (16:35 -0700)]
lib: Split up logic in nfs_flush_chown_uid
This splits up the logic of nfs_flush_chown_uid() so that there is an
individual chown call for each platform instead of a single call that is
compiled for both platforms.
This is the first step to adjusting how nfs attributes are flushed on
Linux because of an issue with chown resetting permissions on the files.
As a example for an simple tcp health check the health-check.sh script
is added. It closes the connection after 10 seconds and only answer to
"PING\n" with a "PONG\n". It is installed by default next to the script
executable.
Markus Valentin [Wed, 19 Jun 2019 12:36:12 +0000 (14:36 +0200)]
script: add a parameter -p for passthrough scripts
Adding the passthrough option to script, this allows to configure a
script to be straightly called from a socket connection, without
implementing any specific protocol.
Timo Sirainen [Mon, 10 Jun 2019 20:07:56 +0000 (23:07 +0300)]
lib-index: Compress cache immediately after enough mails have been expunged
This fixes a bug where cache may never become compressed in certain mailbox
access patterns. Especially for autoexpunging in lazy_expunge mailboxes.
The cache compression happened only when:
a) After the mailbox_sync() that finishes expunging there was another
mailbox_sync(). If mailbox was immediately closed after expunging, this
didn't happen.
b) Fields are fetched from cache
If neither of these happened, the cache just kept growing. This happened
especially with lazy_expunge mailboxes.
sdbox format was always doing b) during expunging, because it looked up
GUIDs from cache. However, this helped only with regular expunges, not
with autoexpunges, because autoexpunging didn't finish with any
mailbox_sync() (which is a bug on its own).
mdbox also did GUID lookups from cache, but it wasn't doing cache
compressions due the bug fixed by the previous commit.
Timo Sirainen [Mon, 10 Jun 2019 20:01:16 +0000 (23:01 +0300)]
lib-index: Fix calling expunge handlers with mdbox
This fixes updating deleted_record_count in dovecot.index.cache files.
Because they were wrong, the cache wasn't always compressed as early as
it should have been.
mdbox uses a separate transaction to commit expunges while mailbox is being
synced. When syncing was finished, tail_offset was updated too early so
mail_index_map(MAIL_INDEX_SYNC_HANDLER_FILE) was always a no-op and expunge
handlers were never called.
There doesn't seem to be any downside to not updating tail_offset early.
sdbox saving also uses such a transaction, but there is no difference in
the resulting dovecot.index.log file. The main worry I had was that
tail_offset wouldn't be updated to point to the end of the log file.
However, this doesn't happen with the old code either. This is because the
extra transaction is external, and tail_offset updating skips over all
external transactions anyway.
This allows accessing the global mail storage service. Normally there is
only a single service created for the process at a time, so this allows
accessing it.
Markus Valentin [Thu, 13 Jun 2019 18:54:39 +0000 (20:54 +0200)]
lmtp-proxy: set real_ variables in auth_user_info struct
Set the real_ variables from connection to the client struct and from
there to the auth_user_info. This allows to pass the real_ variables to
lib-auth and thereby to the auth process.
Markus Valentin [Thu, 13 Jun 2019 18:45:55 +0000 (20:45 +0200)]
lib-auth: add real_[remote|local]_[ip|port] to auth_user_info
This allows the auth_user_info_export function to export all the real_
variables, if available, to the auth request thereby enableing real_
variables for lib-auth.
stats: event export - Increase default http-post timeout to 250ms
50ms just isn't enough. A slightly higher latency link combined with a
handful of storage I/Os can easily use up 50ms.
If each event has approximately 4kB memory footprint, then a 250ms timeout
keeps the memory requirement relatively low (50MB on average / 200MB max at
50k events/sec) but still allows a decent amount of time for the HTTP POST
to get sent to and processed by a server in the same datacenter.
Timo Sirainen [Wed, 12 Jun 2019 07:58:37 +0000 (10:58 +0300)]
lib-storage: Sync mailbox after autoexpunging
This actually finishes expunging the mails. Previously the mails were
just marked to be expunged in the transaction log, and the next session
that opened the mailbox finished the expunging.
Timo Sirainen [Tue, 5 Feb 2019 22:17:52 +0000 (14:17 -0800)]
lib-index: Make sure tail_offset is updated after calling expunge handlers
Since it's a bit troublesome to track whether expunge handlers were actually
called, we can just assume that as long as syncing sees expunges, they were
called (and thet most likely were).
This fixes a bug where dovecot.index.cache header's record_count kept
shrinking and deleted_record_count kept increasing after each sync.
These values were only used to determine when to compress the cache, so
its only effect was that the cache could have been compressed earlier
than necessary.
If header with the same key already exists, just replace the value.
HTTP supports having multiple headers with the same key only when they
can be rewritten into a single comma-separated header. So practically
there's no reason for lib-http to need to support adding multiple
headers. Replacing an existing value is more useful generally.
Aki Tuomi [Fri, 19 Apr 2019 17:39:32 +0000 (20:39 +0300)]
lib-dict: dict-sql - Add username as parameter
SQL statement caching works better when the username is provided
as parameter instead of part of the query, since the caching
can now cache same statements despite username changes.
The previous code always assumed it would be called from another thread, but
due to race conditions this wasn't true always. The lookup could have
already finished by the time cass_future_set_callback() was called, in which
case it called the callback immediately.
If the callback was called from the main thread, it still wrote to the
fd_pipe. Because this was a blocking write, it was possible that the pipe
ended becoming full, which caused the whole dict process to hang.