Tobias Oetiker [Fri, 5 Dec 2008 15:18:11 +0000 (15:18 +0000)]
check for broken implementations of msync / MS_ASYNC where
the mtime does not get updated. warn and explicitly call utime
in rrd_open when a file is opened rw.
Tobias Oetiker [Wed, 3 Dec 2008 11:08:48 +0000 (11:08 +0000)]
When fetching data from 'past' an rra, rrd_fetch was seeking past the rrd
file. This did cause seek errors in some setups. Fixed. Thanks to Daniel Pocock for finding the problem.
Tobias Oetiker [Thu, 20 Nov 2008 13:04:38 +0000 (13:04 +0000)]
Attached a patch for the rrd-tool LIBDBI integration with the following improvements:
a) correct error handling in case of libdbi being unable to load the driver
was producing segmentation faults.
b) better parsing of datasources
* until now timestamp fields had to be integer and had to contain a unix
timestamp - now you can now also use DateTime fields (you still need to
specify it, as the time-range needs to be defined correctly)
* data fields are now no longer limited to (var)char or DOUBLE fields -
FLOAT, INTEGER,... are now also supported.
c) there is a bug with at least LIBDBI 0.8.1 in conjunction with mysql that
can result in segmentation faults when BINARY/BLOB fields are accessed -
rrdtool will now tell you about this fact before dying ;)
d) also the value of rrdderivemaxstep only gets applied if derive has been
selected correctly.
e) "GROUP BY timestamp" has been removed from SQL statement.
f) "ORDER BY timestamp" will be added only in the case of fetching "derived" data.
Tobias Oetiker [Tue, 18 Nov 2008 17:19:17 +0000 (17:19 +0000)]
win32 portability patch and win32/rrdlib.vcproj file for the source
-- christof wegmann Christof.Wegmann with exitgames.com
some fixes to make the result work on unix again ...
Tobias Oetiker [Tue, 11 Nov 2008 16:36:13 +0000 (16:36 +0000)]
This patch allows for multiple RRD writer threads to service the queue.
Now, rrdcached can have more simultaneous I/O requests into the kernel.
This may allow the kernel to re-order disk writes, resulting in better
disk throughput. In practice, throughput is increased by 2-3x.
The flush (-f) timer maintenance has been moved from queue_thread_main
into its own thread.
Be more careful about when to use pthread_cond_signal vs _broadcast, since
multiple threads may be waiting on &queue_cond.
Tobias Oetiker [Fri, 7 Nov 2008 14:07:53 +0000 (14:07 +0000)]
As some of you may know that I have created a patch for rrdtool 1.2 a few years ago, so that a database could be queried for
values for graphing.
The patch has been mostly rewritten and the following changes have been made:
* high dependency on mysql has been reduced by avoiding the
temporary tables (which was bad for mysql replication)
* The number of executed SQL-Statements for one CDEF has been
reduced to 1 compared to 11 SQLs (including CREATE TEMPORARY
TABLE) - for patch against version 1.2
* All consolidation is done in rrdtool itself (MIN,MAX,AVERAGE)
* Additional consolidation functions are COUNT and SIGMA, which give
information on statistics on a per "time-bin" basis.
* All these consolidation values are always returned as separate
columns, that are returned by RRD and the consolidation function
given as Argument is ignored.
Main reason is that this way there is only one call to rrd_fetcht
and thus the database even if we need to fetch for example min,
avg and max. Compare this to 3 calls in case of different
consolidation functions - and if you want to get SIGMA and COUNT
as well it is still only one call to the backend and the database.
* Some previous existing features have been taken out at the moment
to allow for this reduced set of SQL queries.
o prediction using the values from the last X days at the same
time
o the corresponding sigma calculation
* The idea is to create generic CDEF's that will do the same thing,
but that is also available when using RRD-files (similar to TREND,
but with another scope)
This will get posted as a separate patch.
* Overall performance should be much better and the patch as a whole
simpler.
* The patch also includes modifications to the configuration
infrastructure, to make libdbi support optional.
Tobias Oetiker [Fri, 7 Nov 2008 13:51:24 +0000 (13:51 +0000)]
Much simpler handling of timestamp errors. Return an error to the user
when any of the time stamp values are invalid. This is similar to
RRDTool's normal behavior. Removed the complex logic previously used to
return error codes to the user.
This solves a bug where non-advancing timestamps could have produced
incorrect error output during "BATCH" mode. The bug was cause by using
the sock->wbuf pointer for the error output. -- kevin brintnall
Tobias Oetiker [Tue, 4 Nov 2008 07:12:46 +0000 (07:12 +0000)]
I realize now that the problem is the line
test -f lua/Makefile && cd lua && $(MAKE) install || true
in the target "install-data-local", in bindings/Makefile.am. It forces
execution of bindings/lua/Makefile independently of lua being found or
not. I added that line in my first patch, following perl, python and
ruby build style, but it's not needed after I switched to automake. The
make recursion is controlled by SUBDIRS, which will only contain "lua"
if BUILD_LUA is true.
Tobias Oetiker [Tue, 28 Oct 2008 08:57:13 +0000 (08:57 +0000)]
- remove the spaccing between the elements
- add xsd support to dump output
- change the argument "[--no-header|-n]" to [--header|-h {xsd,dtd}]
-- tobias.lindenmann 1und1.de
Tobias Oetiker [Wed, 22 Oct 2008 20:41:59 +0000 (20:41 +0000)]
The previous code relied on the assumption that pthread_cond_init(&cond)
was equivalent to memset(&cond,0). This may not be true on all platforms.
-- kevin
Tobias Oetiker [Wed, 22 Oct 2008 06:02:23 +0000 (06:02 +0000)]
remove_cache_item() did not check whether a file was in queue before
modifying the cache head/tail pointers. Therefore, the process of
flushing old files may perturb the cache_queue_head pointer. This caused
some nodes with CI_FLAGS_IN_QUEUE to be un-linked from the queue list.
Thereafter, they would not be flushed by any periodic process (although
they could be revived with FLUSH or UPDATE). This caused a slow memory
leak for files that are no longer updated. Pending updates for these
"abandoned" files would remain in memory ad infinitum.
With this patch, remove_from_queue() will check that the item is queued
before modifying the head/tail pointers. This restores the intended
behavior.
--kevin
Tobias Oetiker [Tue, 21 Oct 2008 05:42:50 +0000 (05:42 +0000)]
* Open all listen sockets in daemonize(), while we still have stderr.
Changed open_listen_socket_* routines to complain to stderr. Now, any
errors in binding to the listen sockets are much more obvious.
* Simplified exit of parent after fork()
* PID file will be correctly cleaned up if there is a failure in daemonize().
* unlink the unix socket before trying to bind()
(after we're sure we have the PID file)
Tobias Oetiker [Mon, 20 Oct 2008 11:46:08 +0000 (11:46 +0000)]
rrd_notify_row patch:
- Delegate choice of starting row for newly created RRD files to the rrd_open.c API.
- Introduce the rrd_notify_row() function so that an implementation can choose to align the rows of new RRDs with existing RRDs, if desirable.
- Maintain the existing behaviour (random starting row) by default.
Tobias Oetiker [Sat, 18 Oct 2008 22:32:19 +0000 (22:32 +0000)]
rrd_open should not create files with restrictive masks
Removed unnecessary "mode" varaible. The mode is only used when O_CREAT is
specified, where we want to use 0666 (as rrd_create_fn did r<=1612).
--kevin
Tobias Oetiker [Sat, 18 Oct 2008 15:50:07 +0000 (15:50 +0000)]
- encapsulate fd and mmap related variables within a private data
structure
- rrd_file_t keeps a pointer to the private data structure of type
void*, so that other block storage implementations can store their
internal data with rrd_file_t
-- Daniel.Pocock
Tobias Oetiker [Thu, 16 Oct 2008 21:12:27 +0000 (21:12 +0000)]
- rrd_open() calculates file size for new files and calls mmap once for
the whole file
- rrd_resize() cleaned up, no longer passing a size through the cookie
argument
- rrd_init(&my_rrd) must be called before rrd_open() - if people are
calling rrd_open directly from application code, this might be
troublesome. Alternative solutions: creating an additional function,
rrd_open_create(), or adding an extra argument to rrd_open() for setting
the file size
Tobias Oetiker [Tue, 14 Oct 2008 20:14:35 +0000 (20:14 +0000)]
This moves selection of the initial RRA row into the rrd_open.c API
The current implementation (random row) is used by default. However, it
now provides an opportunity for alternative implementations to integrate
with rrdtool in a single place.
Maybe there are other places in rrdtool where I should insert calls to
the function rrd_notify_row()?
This has been tested with rrdtool create and rrdtool info to verify that
random rows are selected by default (existing behaviour preserved).
Tobias Oetiker [Tue, 14 Oct 2008 19:23:24 +0000 (19:23 +0000)]
Under most circumstances, rrdcached can detect a stale pid file.
If the process in the pid file does not exist, or cannot be signalled by
the rrdcached owner, then rrdcached will replace the pid file and start
normally. Otherwise, it will complain verbosely to STDERR.
Tobias Oetiker [Tue, 14 Oct 2008 19:08:56 +0000 (19:08 +0000)]
* this preserves principle of least surprise when dealing with files that
are reachable via many path strings. i.e. when $PWD=/base/dir the
following files are the same:
/base/dir/x.rrd
x.rrd
../dir/x.rrd
* for performance, absolute paths (starting with '/') are not resolved.
this reduces the number of stat(2) system calls.
Tobias Oetiker [Mon, 13 Oct 2008 22:07:14 +0000 (22:07 +0000)]
This patch reduces the number of time()/gettimeofday() system calls when
doing high volume processing. This enables about 25% speed increase
during journal replay and "BATCH" processing. (this is a function of
syscall overhead).
* note when "BATCH" processing or journal replay starts, use that
timestamp for all commands
* use the batch start time to detect when we're in batch mode. no longer
need a separate boolean.
* pass the time_t into handle_request
* pass the time_t through to the commands that need it
Tobias Oetiker [Sat, 11 Oct 2008 09:37:53 +0000 (09:37 +0000)]
This patch introduces a feature whereby rrdcached will disallow updates
that do not advance the update time. This prevents the updates from being
discarded later by rrd_update_r.
This patch attempts to make the most of the protocol's limited ability to
return error text when using a -1 return code.
Tobias Oetiker [Fri, 10 Oct 2008 05:21:19 +0000 (05:21 +0000)]
This patch ensures that when rrdcached is stopped, it cleans up the
pid file. Apparently this is necessary if RRDCACHED_USER is not the
default "rrdcached".
-- Bernard Li
Tobias Oetiker [Fri, 10 Oct 2008 05:21:01 +0000 (05:21 +0000)]
The patch I submitted for rrdtool.spec introduced a bug where there
are two ldconfig calls in the %postun section. This patch fixes that.
-- Bernard Li
Tobias Oetiker [Tue, 7 Oct 2008 21:08:30 +0000 (21:08 +0000)]
This patch moves the permission handling code around a bit.
* moved privilege checks into the command handler functions
(possible now that we pass the sock data structures around)
* on UPDATE, delay journal_write until after check_file_access().
previously, it was possible for a high-priv socket to introduce
commands into the journal that could be replayed if they were
still in the journal at next startup.
* moved has_privilege() further up in the file to avoid need
for prototype.
Tobias Oetiker [Tue, 7 Oct 2008 16:28:24 +0000 (16:28 +0000)]
This patch introduces some extra safety checks in journal processing,
and cleans up the code a little bit.
* moved journal initialization to its own function; main() is cleaner
* any time we process a file, log the results
(previous code only loggded if there was a valid entry)
* After reading journals at startup, only trigger full flush out to disk
if the user specified -F. Avoids unnecessary IO on startup unless the
user also wants unnecessary IO on shutdown.
* journal_replay is much more careful about files it will open
* must be a regular file
* must be owned by daemon user
* must not be group/other writable
* Ensure that the journal gets created with the right permissions.
... even when the daemon is invoked with a permissive umask.
equivalent to "chmod a-x,go-w"
Tobias Oetiker [Tue, 7 Oct 2008 15:37:34 +0000 (15:37 +0000)]
aniel Pocock reported that the argument may be NULL in low-diskspace
situations, so check for that here to prevent a segmentation fault.
-- Florian Forster
Tobias Oetiker [Mon, 6 Oct 2008 19:05:47 +0000 (19:05 +0000)]
This patch introduces "BATCH" mode.
In this mode, a client can feed multiple commands to rrdcached without
waiting for acknowledgement. This permits multiple commands to be sent
for each read()/write(). This can dramatically increase the command
throughput by increasing the amount of work done per system call.
It enables over 100k updates/second with no CPU
utilization due to the reduced system calls.
Tobias Oetiker [Mon, 6 Oct 2008 19:04:48 +0000 (19:04 +0000)]
This patch introduces buffered I/O to rrdcached. Now, rrdcached can
interpret as many commands as arrive in a single read(), and it will use
fewer write()s when there are multiple output lines.
All routines now pass around listen_socket_t objects instead of file
descriptors.
All I/O is now contained in two routines. It's no longer necessary to
specify the line count in multi-line outputs, since that is calculated
automatically.
This is the foundation for accepting batched commands.
-- kevin brintnall
Tobias Oetiker [Wed, 1 Oct 2008 20:22:57 +0000 (20:22 +0000)]
since rrdcached uses pthread functions, use the threadsafe version of librrd as well. This will
also reasolve build problems on boxes there the ptherad functions must be linked explicitly.
projects / thirdparty / rrdtool-1.x.git / log
