Ander Juaristi [Fri, 26 Apr 2019 07:58:07 +0000 (09:58 +0200)]
IPFIX: Introduce template record support
This commit adds the ability to send template records
to the remote collector.
In addition, it also introduces a new
configuration parameter 'send_template', which tells when template
records should be sent. It accepts the following string values:
- "once": Send the template record only the first time (might be coalesced
with data records).
- "always": Send the template record always, with every data record that is sent
to the collector (multiple data records might be sent together).
- "never": Assume the collector knows the schema already. Do not send template records.
If omitted, the default value for 'send_template' is "once".
Signed-off-by: Ander Juaristi <a@juaristi.eus> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
I am currently testing it by launching a plain NetCat listener on port
4739 (the default for IPFIX) and then running Wireshark and see that it
dissects the IPFIX/NetFlow traffic correctly (obviously this relies on
the Wireshark NetFlow dissector being correct).
First:
nc -vvvv -l 127.0.0.1 4739
Then:
sudo ulogd -vc ulogd.conf
Signed-off-by: Ander Juaristi <a@juaristi.eus> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
These files are outdated and they belong to downstream users (distributions).
Providing outdated and unmaintained files here serves no purpose other than
confusing users and annoy packagers.
If an user is using ulogd2 directly from the source tarball, I would expect it
to be proficient enough to generate these files by itself.
Signed-off-by: Arturo Borrero Gonzalez <arturo@netfilter.org> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
This new configuration behaviour option eases a bit the configuration of ulogd2
by allowing to load all plugins in one go, without having to know their full
path.
Choosing concrete plugins and using full path for them is great for some
environmnets, but I don't think it's a common case. The common case is to
load all plugins, even ignoring where do they live in the filesystem.
Even worse, the full path may be architecture-dependant, which makes copying
the ulogd.conf file between machines unnecesarily complex.
To experiment this new behaviour, don't put any 'plugin=' directive in the
config file. Plugins will be loaded from a default directory, choosen at
build/configure time (--with-ulogd2libdir). If no specified, this is something
like '/usr/local/lib/ulogd/'.
This new configuration option doesn't implement any special logic. We simply
open the dir and try to load all files ending with '.so'.
The log message level for plugins loading is increased so users can see by
default which plugins are loaded.
Signed-off-by: Arturo Borrero Gonzalez <arturo@netfilter.org> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
Is common that ulogd runs in scenarios where a lot of packets are to be logged.
If there are more packets than ulogd can handle, users can start seing log
messages like this:
ulogd[556]: We are losing events. Please, consider using the clauses \
`netlink_socket_buffer_size' and `netlink_socket_buffer_maxsize'
Which means that Netlink buffer overrun have happened.
There are several approaches to prevent this situation:
* in the ruleset, limit the amount of packet queued for log
* in the ruleset, instruct the kernel to use a queue-threshold
* from userspace, increment Netlink buffer sizes
* from userspace, configure ulogd to run as high priority process
The first 3 method can be configured by users at runtime.
This patch deals with the last method. SCHED_RR is configured by default,
with no associated configuration parameter for users, since I believe
this is common enough, and should produce no harm.
A similar approach is used in the conntrackd daemon.
Signed-off-by: Arturo Borrero Gonzalez <arturo@netfilter.org> Acked-by: Eric Leblond <eric@regit.org> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
Eric Leblond [Sun, 2 Jul 2017 15:57:54 +0000 (17:57 +0200)]
ip2bin: fix plugin link for some compiler
Declaring a function inline and building with -O0 was causing the
following message:
undefined symbol: uint32_to_ipv6
By declaring the function as static we fix the problem.
Liping Zhang [Tue, 11 Oct 2016 14:22:27 +0000 (22:22 +0800)]
ulogd: fix crash when ipv4 packet is truncated
If ipv4 packet is truncated, we should not try to dereference the
iph pointer. Otherwise, if the user add such iptables rules
"-j NFLOG --nflog-size 0", we will dereference the NULL pointer
and crash may happen.
Reported-by: Chris Caputo <ccaputo@alt.net> Signed-off-by: Liping Zhang <liping.zhang@spreadtrum.com> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
Helmut Schaa [Fri, 18 Mar 2016 14:43:24 +0000 (15:43 +0100)]
ulogd: fix cross compilation errors with mysql_config
When cross-compiling ulogd, mysql_config and pg_config will return build host
configuration not build target configuration. This leads to build failures
if mysql_config is installed on the host system but mysql is not available
on the build target.
Fix this by not using mysql_config and pg_config for cross-compilation.
Signed-off-by: Helmut Schaa <helmut.schaa@googlemail.com>
Hironobu Ishii [Tue, 2 Feb 2016 14:01:41 +0000 (23:01 +0900)]
ulogd: restructures signal handling by self-pipe trick
ulogd had a critical bug that is calling Async-Signal-Unsafe functions
in signal hander context.
- Most of libc functions like fopen(), malloc() are Async-Signal-Unsafe.
So you should not call these functions in signal handler context.
- Calling pluginstances in signal handler context is danger.
For implementer of pluginstances, it is very hard to recognize their
functions are called in signal handler context.
To solve the issue, I restructured signal handling by self-pipe trick.
For more detail on self-pipe trick, please see the following.
https://lwn.net/Articles/177897/
This patch will solve various symptoms like following.
- Deadlock
- Segmentation fault caused by libc management data corruption,
- Other unpredictable behavior.
Deadlock example
================
This bug was already filed at:
https://bugzilla.netfilter.org/show_bug.cgi?id=1030
I also hit this bug. The backtrace of this issue is following.
In this case, main thread was calling ctime(),
and signal handler called localtime_r().
That caused the dead lock while getting tzset_lock in __tz_convert().
Because vsyslog() is Async-Signal-Unsafe function, we cannot call
this function in signal handler context.
(gdb) bt
#0 __lll_lock_wait_private () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
#1 0x00007f3c3fc7e4ac in _L_lock_2462 () at tzset.c:621
#2 0x00007f3c3fc7e2e7 in __tz_convert (timer=0x7f3c3ff8bf00 <tzset_lock>,
timer@entry=0x7fffcfa923b8, use_localtime=use_localtime@entry=1,
tp=tp@entry=0x7fffcfa92400) at tzset.c:624
#3 0x00007f3c3fc7c28d in __localtime_r (t=t@entry=0x7fffcfa923b8,
tp=tp@entry=0x7fffcfa92400) at localtime.c:32
#4 0x00007f3c3fcbf1ba in __GI___vsyslog_chk (pri=<optimized out>, flag=1,
fmt=0x406fa8 "signal received, calling pluginstances\n", ap=0x7fffcfa924a0)
at ../misc/syslog.c:199
#5 0x00000000004037b5 in __ulogd_log ()
#6 0x00000000004047be in signal_handler ()
#7 <signal handler called>
#8 0x00007f3c3fcb62f5 in __GI___xstat (vers=<optimized out>,
name=0x7f3c3fd4b2c3 "/etc/localtime", buf=0x7fffcfa92c10)
at ../sysdeps/unix/sysv/linux/wordsize-64/xstat.c:37
#9 0x00007f3c3fc7e5f6 in __tzfile_read (file=file@entry=0x7f3c3fd4b2c3 "/etc/localtime",
extra=extra@entry=0, extrap=extrap@entry=0x0) at tzfile.c:170
#10 0x00007f3c3fc7d954 in tzset_internal (always=<optimized out>,
explicit=explicit@entry=1) at tzset.c:444
#11 0x00007f3c3fc7e303 in __tz_convert (timer=0x7fffcfa92d50,
use_localtime=use_localtime@entry=1, tp=tp@entry=0x7f3c3ff8ed80 <_tmbuf>)
at tzset.c:629
#12 0x00007f3c3fc7c2a1 in __GI_localtime (t=<optimized out>) at localtime.c:42
#13 0x00007f3c3fc7c1f9 in ctime (t=<optimized out>) at ctime.c:27
#14 0x00007f3c3e180ec2 in ?? ()
#15 0x0000000056a100c2 in ?? ()
#16 0xf8570f79d4fc4200 in ?? ()
#17 0x000000000209bec0 in ?? ()
#18 0x00007f3c4059f1f8 in ?? ()
#19 0x000000000000003c in ?? ()
#20 0x0000000000404952 in ulogd_propagate_results ()
#21 0x00007f3c3f9cc203 in ?? ()
#22 0x0000000000000000 in ?? ()
Segmentation fault in free()
============================
>From my experience, I think this was caused by some routine called
malloc()/free() in signal handler context.
By that, malloc() management data became inconsistent.
As a result, free() made a wrong dereference.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 __GI___libc_free (mem=0x7f430f011000) at malloc.c:2903
2903 if (chunk_is_mmapped(p)) /* release mmapped memory. */
(gdb) bt
#0 __GI___libc_free (mem=0x7f430f011000) at malloc.c:2903
#1 0x00007f430e68affa in __GI__IO_free_backup_area (fp=fp@entry=0x742500)
at genops.c:210
#2 0x00007f430e68a795 in _IO_new_file_overflow (f=0x742500, ch=-1) at fileops.c:849
#3 0x00007f430e689511 in _IO_new_file_xsputn (f=0x742500, data=<optimized out>, n=15)
at fileops.c:1372
#4 0x00007f430e65aa4d in _IO_vfprintf_internal (s=s@entry=0x742500,
format=<optimized out>, format@entry=0x7f430cbc4008 "%.15s %s %s",
ap=ap@entry=0x7fff456ece38) at vfprintf.c:1635
#5 0x00007f430e71d615 in ___fprintf_chk (fp=0x742500, flag=flag@entry=1,
format=format@entry=0x7f430cbc4008 "%.15s %s %s") at fprintf_chk.c:36
#6 0x00007f430cbc3f04 in fprintf (__fmt=0x7f430cbc4008 "%.15s %s %s",
__stream=<optimized out>) at /usr/include/bits/stdio2.h:97
#7 _output_logemu (upi=0x74e5a0) at ulogd_output_LOGEMU.c:102
#8 0x0000000000404952 in ulogd_propagate_results ()
#9 0x00007f430e40f203 in interp_packet (ldata=0x7fff456ed060, pf_family=2 '\002',
upi=0x74a6b0) at ulogd_inppkt_NFLOG.c:400
#10 msg_cb (gh=<optimized out>, nfmsg=0x7f430efe2020, nfa=0x7fff456ed060, data=0x74a6b0)
at ulogd_inppkt_NFLOG.c:483
#11 0x00007f430e20a307 in __nflog_rcv_pkt (nlh=<optimized out>, nfa=<optimized out>,
data=<optimized out>) at libnetfilter_log.c:160
#12 0x00007f430e0056b7 in __nfnl_handle_msg (len=268, nlh=0x7f430efe2010, h=0x74e8e0)
at libnfnetlink.c:1236
#13 nfnl_handle_packet (h=0x74e8e0, buf=0x7f430efe2010 "\f\001", len=<optimized out>)
at libnfnetlink.c:1256
#14 0x00007f430e20a508 in nflog_handle_packet (h=<optimized out>, buf=<optimized out>,
len=<optimized out>) at libnetfilter_log.c:323
#15 0x00007f430e40eaed in nful_read_cb (fd=<optimized out>, what=<optimized out>,
param=0x74a6b0) at ulogd_inppkt_NFLOG.c:463
#16 0x0000000000404ee0 in ulogd_select_main ()
#17 0x0000000000402b17 in main ()
Vincent Bernat [Fri, 2 Oct 2015 15:00:58 +0000 (17:00 +0200)]
json: append timezone information to ISO 8601 date
While this is not strictly needed for ISO 8601, this is helpful since
otherwise, the receiver can't assume anything about the
timezone.
This uses a GNU extension but as ulogd is quite Linux-specific, this
shouldn't be a problem. The POSIX variables (tzname and daylight) are
quite difficult to use because daylight handling is incomplete (daylight
don't say if DST is now in effect, it just says it is sometimes in
effect).
A timezone offset is used instead of a timezone since it is usually
easier to parse (strptime in glibc is not able to parse a timezone name)
and don't require an up-to-date TZ database.
Signed-off-by: Vincent Bernat <Vincent.Bernat@exoscale.ch>
Vincent Bernat [Wed, 30 Sep 2015 12:32:07 +0000 (14:32 +0200)]
json: output messages in JSONv1 format
While Logstash is quite flexible in the JSON messages received, the
canonical format it "expects" is the JSON Event v1 format. The timestamp
should be keyed by `@timestamp` and there should be a `@version` key
whose value is 1. All other keys are free.
There is no formal specification of this format. It is however described
here:
It's useful to respect this format as it allows a user to use a less
capable receiver. The new format is enabled only when `eventv1=1` is set
in plugin configuration.
Signed-off-by: Vincent Bernat <Vincent.Bernat@exoscale.ch>
Felix Janda [Sat, 16 May 2015 13:44:32 +0000 (15:44 +0200)]
Define _GNU_SOURCE to get members of tcphdr
The source uses linux names for members of tcphdr. For example
"source" instead of "th_sport", ... musl libc's headers need
_GNU_SOURCE defined in order to expose these.
Signed-off-by: Felix Janda <felix.janda@posteo.de>
Eric Leblond [Fri, 7 Mar 2014 20:11:47 +0000 (21:11 +0100)]
json: use packet timestamp if available
This patch updates the JSON output plugin to have it use the
timestamp of the packet if available. The date format used
for the timestamp is now using ISO 8601 to have an easy
import in most software (tested with logstash and splunk).
Eric Leblond [Sat, 22 Feb 2014 13:05:56 +0000 (14:05 +0100)]
nfct: use start timestamp if provided
When hash table is not used, the start timestamp was not used even
if the kernel is sending it. This patch modifies the code to use it
when available. This allows to log connection with start and end
with a single message per connection and without the cost of
maintaining the hash table.
Eric Leblond [Tue, 4 Feb 2014 08:27:45 +0000 (09:27 +0100)]
ulogd: add carriage return as separator
If the file is in DOS mode, the string coming from config file
parsing are containing the carriage return. The result is that
string are not correct and the parsing of confuguration file
is failling.
Eric Leblond [Sun, 26 Jan 2014 16:05:29 +0000 (17:05 +0100)]
json: introduce new JSON output plugin
This patch introduces a new JSON output plugin. This
patch displays CIM field name instead of ulogd key valu
if this CIM field is available.
The module does not display binary address but uses the
string version of them. So a complete stack is for example:
stack=log2:NFLOG,base1:BASE,ifi1:IFINDEX,ip2str1:IP2STR,mac2str1:HWHDR,json1:JSON
If boolean_label is set to 1, then the numeric_label put on packet
by the input plugin is coding the decision on packet. If 0, then
packet has been blocked and if non null it has been accepted.
Eric Leblond [Sun, 26 Jan 2014 21:21:42 +0000 (22:21 +0100)]
store Common Information Model name in ulogd key
This patch adds storage for CIM field name in ulogd key. This
will be used by JSON output to interoperate with logging
collector such as logstash or splunk.
Common Information Model is an open standard that defines how managed
elements in an IT environment are represented as a common set of objects
and relationships between them:
http://www.dmtf.org/standards/cim
This seems to be mainly XML based but there is a JSON version of some
aspects of the model. One of the main documentation on CIM in JSON
format seems to be:
http://docs.splunk.com/Documentation/PCI/2.0/DataSource/CommonInformationModelFieldReference
Using the correct CIM field name allow events coming from ulogd to be
correlated with events coming from other sources.
Eric Leblond [Sat, 25 Jan 2014 11:51:49 +0000 (12:51 +0100)]
db: cancel injection thread when terminating
Injection thread was not cancelled when a termination signal was
sent. This was causing a crash in some cases. This patch fixes this
by canceling the thread when a SIGTERM or SIGINT signal is received.
Eric Leblond [Sat, 25 Jan 2014 11:26:38 +0000 (12:26 +0100)]
db: set ring default size to 0
As default size was non null, the ring system was activated by
default. It was only possible to desactivate the ring system
by setting it to ring_buffer_size to 0 in the configuration.
This was not the attended behavior. This patch set it to 0 to have
only explicit activation of the ring feature.
Eric Leblond [Sat, 25 Jan 2014 10:46:52 +0000 (11:46 +0100)]
util: fix warning on format string
pp_print macro was not correctly formatting u64. This patch renames
it to pp_print_u as it is only used to print integer. It also use
the PRIu* macros to have a correct format string for all integers
type.
Gustavo Zacarias [Tue, 10 Dec 2013 12:24:12 +0000 (09:24 -0300)]
ulogd: use AC_SEARCH_LIBS for libpthread
Some uClibc-based toolchains lack threading support, so use
AC_SEARCH_LIB instead of AC_CHECK_LIB to check for libpthread
availability and link conditionally if found since it's only used for
the database backends.
Chris Boot [Mon, 18 Nov 2013 11:09:14 +0000 (11:09 +0000)]
ulogd: ulogd_inppkt_NFLOG: close nflog handle after unbinding
The nflog handle is closed, and then nflog_unbind_pf() is called on it, which
triggers an assertion failure within libnfnetlink. This patch simply moves
the nflog_close() just after the nflog_unbind_pf() calls.
Eric Leblond [Tue, 28 May 2013 19:58:57 +0000 (21:58 +0200)]
pgsql: add var to specify arbitrary conn params
This patch adds a configuration variable for PostgreSQL output.
Named connstring it stores the character string that will be
used to connect to the PostgreSQL server. This allows the user
to use all options available like TLS parameters for example.
Eric Leblond [Sat, 18 May 2013 13:25:11 +0000 (15:25 +0200)]
Improve pid file handling.
This patch improves latest patch by splitting in two part the pid
file creation. This allows to display a message to stdout when
ulogd can not be started. Another linked improvement is that the
plugin initialization is not done if the pid file existence will
result in a ulogd exit.
Chris Boot [Sun, 12 May 2013 12:47:40 +0000 (13:47 +0100)]
ulogd: Implement PID file writing
The deamon currently does not have the ability to write a PID file to track its
process ID. This is very useful to an init script and to ensure there is only
one running instance. This patch implements this functionality.
Chris Boot [Sat, 11 May 2013 17:01:52 +0000 (18:01 +0100)]
ulogd: Perform nice() before giving up root
The daemon code currently tries to nice(-1) just after having given up root
privileges, which fails. This patch moves the nice(-1) call to just before
the code that gives up the required privileges.
Victor Julien [Fri, 5 Apr 2013 16:28:14 +0000 (18:28 +0200)]
Exec libmnl config check only if nfacct is enabled
In case nfacct is not enabled in ulogd2, libmnl is not used. So it shouldn't
be a hard global dependency, but instead only a dependency in case nfacct is
enabled.
Eric Leblond [Thu, 9 May 2013 09:20:37 +0000 (11:20 +0200)]
db: disable SIGHUP if ring buffer is used.
The handling of signal when using threads can be complicated. When
ring buffer is used for query, this means ulogd will have to follow
some sort of mutex. Thus, it is easier and better performance wise
to disable the reload via SIGHUP when the ring buffer is used.
Eric Leblond [Sat, 20 Apr 2013 10:44:17 +0000 (12:44 +0200)]
db: add ring buffer for DB query
This patch adds an optional ring buffer option which modify
the way database queries are made. The main thread is only handling
kernel message reading and query formatting. The SQL request is made
in a separate dedicated thread.
The idea is to try to avoid buffer overrun by minimizing the time
requested to treat kernel message. Doing synchronous SQL request, as
it was made before was causing a delay which could cause some messages
to be lost in case of burst from kernel side.
Eric Leblond [Sun, 17 Mar 2013 18:41:36 +0000 (19:41 +0100)]
db: store data in memory during database downtime
This patch is adding a mechanism to store query in a backlog build
in memory. This allow to store events during downtime in memory and
realize the effective insertion when the database comes back.
A memory cap is used to avoid any memory flooding.
Ilya Tumaykin [Wed, 20 Mar 2013 10:54:36 +0000 (14:54 +0400)]
Fix automagic support of dbi, pcap and sqlite3
ulogd has automagic deps for several output plugins right now, namely dbi,
pcap and sqlite3. These plugins are built if the appropriate libs are present
on user's system. While this situation is fine with binary distros it is not OK
on source-based ones such as Gentoo.
The problem arises when such a program links against libs without user's
request and libs are later removed from system which leaves program in a
broken state.
This patch is modifying configure.ac which we apply in our package and which
fixes mentioned issue. It adds 3 new configure options: --
without-{dbi,pcap.sqlite}. I would like to emphasize that this patch doesn't
change default behaviour of configure script at all, so all other distros won't
suffer. We simply add options to explicitly disable any attempts to try and
detect libs for automagic deps, which is enough to avoid unnecessary linkage.
Eric Leblond [Fri, 18 Jan 2013 18:41:19 +0000 (19:41 +0100)]
conf: add flag to allow option setup tuning
This patch adds a flag to the config_entry structure to be able to
tune setup. First usage is to ask config parser not to update a key
if it has been already set.
projects / thirdparty / ulogd2.git / log
