Jason Ish [Mon, 26 Jun 2017 17:04:46 +0000 (11:04 -0600)]
log: wrap rotation and write in lock
The application log is subject to rotation, so the check for
rotation, the actual rotation and write needs to be done under
lock to ensure the file pointer is in a consisten state
at the time of write().
Jason Ish [Tue, 27 Jun 2017 22:47:23 +0000 (16:47 -0600)]
rust/dns: handle multiple txt strings
Fix handling of TXT records when there are multiple strings
in a single TXT record. For now, conform to the C implementation
where an answer record is created for each string in a single
txt record.
Also removes the data_len field from the answer entry. In Rust,
the length is available from actual data, which after decoding
may actually be different than the encoded data length, so just
use the length from the actual data.
Victor Julien [Tue, 27 Jun 2017 13:07:40 +0000 (15:07 +0200)]
pcap: fix linktype raw issues
On OpenBSD 6.0 and 6.1 the following pcap gets a datalink type of
101 instead of our defined DLT_RAW.
File type: Wireshark/tcpdump/... - pcap
File encapsulation: Raw IP
File timestamp precision: microseconds (6)
Packet size limit: file hdr: 262144 bytes
Number of packets: 23
File size: 11 kB
Data size: 11 kB
Capture duration: 7,424945 seconds
First packet time: 2017-05-25 21:59:31,957953
Last packet time: 2017-05-25 21:59:39,382898
Data byte rate: 1536 bytes/s
Data bit rate: 12 kbps
Average packet size: 496,00 bytes
Average packet rate: 3 packets/s
SHA1: 120cff9878b93ac74b68fb9216027bef3b3c018f
RIPEMD160: 35fa287bf30d8be8b8654abfe26e8d3883262e8e
MD5: 13fe4bc50fe09bdd38f07739bd1ff0f0
Strict time order: True
Number of interfaces in file: 1
Interface #0 info:
Encapsulation = Raw IP (7/101 - rawip)
Capture length = 262144
Time precision = microseconds (6)
Time ticks per second = 1000000
Number of stat entries = 0
Number of packets = 23
On Linux it is 12.
On the tcpdump/libpcap site the DLT_RAW is defined as 101:
http://www.tcpdump.org/linktypes.html
Strangely, on OpenBSD the DLT_RAW macro is defined as 14 as expected.
So for some reason, libpcap on OpenBSD uses 101 which seems to match
the tcpdump/libpcap documentation.
So this patch adds support for datalink 101 as RAW.
Victor Julien [Tue, 27 Jun 2017 07:59:48 +0000 (09:59 +0200)]
detect: fix crash when stream inspect runs on UDP
Certain rules can apply to both TCP and UDP. For example 'alert dns'
rules are inspected against both TCP and UDP. This lead to the
stream inspect engine being called on a UDP packet.
This patch fixes the issue by exiting early from the stream inspect
engine if a) proto is not TCP or b) ssn is not available
Eric Leblond [Wed, 21 Jun 2017 17:50:11 +0000 (19:50 +0200)]
output-json-alert: add app_proto or flow to events
This patch adds a partial flow entry in the alert event
(if applayer or flow is selected) or simply app_proto if
it is not.
app_proto is useful as filter and aggregation field. And
the partial flow entry contains more information about the
proto as well as some volumetry info.
Jason Ish [Tue, 20 Jun 2017 16:17:54 +0000 (10:17 -0600)]
rust: save cargo and CARGO_HOME to variables
During configure, substitute the path of cargo, as well as the
value of CARGO_HOME as variables. This fixes the case where a
user might do:
make
sudo make install
Which will cause the cargo bits to be rebuilt, including
re-downloading external crates.
By saving these to variables we can be sure that the same
values are used during make install as were used during
make which prevents the Rust artifacts from being rebuild
during "sudo make install".
Victor Julien [Tue, 20 Jun 2017 10:13:14 +0000 (12:13 +0200)]
tunnel: refactor tunnel verdict handling
Observed:
STARTTLS creates 2 pseudo packets which are tied to a real packet.
TPR (tunnel packet ref) counter increased to 2.
Pseudo 1: goes through 'verdict', increments 'ready to verdict' to 1.
Packet pool return code frees this packet and decrements TPR in root
to 1. RTV counter not changed. So both are now 1.
Pseudo 2: verdict code sees RTV == TPR, so verdict is set based on
pseudo packet. This is too soon. Packet pool return code frees this
packet and decrements TPR in root to 0.
Real packet: TRP is 0 so set verdict on this packet. As verdict was
already set, NFQ reports an issue.
The decrementing of TPR doesn't seem to make sense as RTV is not
updated.
Solution:
This patch refactors the ref count and verdict count logic. The beef
is now handled in the generic function TmqhOutputPacketpool(). NFQ
and IPFW call a utility function VerdictTunnelPacket to see if they
need to verdict a packet.
Remove some unused macro's for managing these counters.
Jason Ish [Fri, 16 Jun 2017 05:38:04 +0000 (23:38 -0600)]
rust/json: only output printable characters
Rust strings are UTF8 and we cannot yet rely on jansson
having json_stringn on all supported OS distributions yet
so sanitize strings to ascii before printing.
Also add set_string_from_bytes which is like set_string, but
accepts a byte array as input.
Victor Julien [Thu, 15 Jun 2017 13:00:12 +0000 (15:00 +0200)]
nfs: rename nfs3 to nfs
Since the parser now also does nfs2, the name nfs3 became confusing.
As it's still in beta, we can rename so this patch renames all 'nfs3'
logic to simply 'nfs'.
Victor Julien [Thu, 8 Jun 2017 13:13:49 +0000 (15:13 +0200)]
rust/nfs: handle GAPs
In normal records it will try to continue parsing.
GAP 'data' will be passed to file api as '0's. New call is used
so that the file API does know it is dealing with a GAP. Such
files are flagged as truncated at the end of the file and no
checksums are calculated.
Eric Leblond [Tue, 13 Jun 2017 20:32:50 +0000 (22:32 +0200)]
bypass: add explicit flag in stream engine
TCP reassembly is now deactivated more frequently and triggering a
bypass on it is resulting in missing some alerts due forgetting
about packet based signature.
So this patch is introducing a dedicated flag that can be set in
the app layer and transmitted in the streaming to trigger bypass.
It is currently used by the SSL app layer to trigger bypass when
the stream becomes encrypted.
Jason Ish [Wed, 7 Jun 2017 19:25:16 +0000 (13:25 -0600)]
rust: don't fail distcheck if cargo-vendor not found
Allow distcheck to pass if cargo vendor is not found by not
failing out. It is not required to successfully build a dist
tarball, the Rust sources will just not be vendored in.
Also don't fail out make dist if Python is not installed. A build
will still be successful is Python is available on the end
build system.
Eric Leblond [Sun, 25 Dec 2016 22:04:31 +0000 (23:04 +0100)]
alert-prelude: correctly set Source and Target
IDMEF alert contains two entities named Source and Target that are
defined using common language:
* "The Source class contains information about the possible source(s) of
the event(s) that generated an alert."
* "The Target class contains information about the possible target(s) of
the event(s) that generated an alert."
Previous alerts event were not following that so we can updated the code
when we know the direction thanks to the metadata field.
Eric Leblond [Fri, 16 Dec 2016 18:26:58 +0000 (19:26 +0100)]
detect-target: introduce new keyword
The target keyword allows rules writer to specify information about
target of the attack. Using this keyword in a signature causes
some fields to be added in the EVE output. It also fixes ambiguity
in the Prelude output.
Jason Ish [Tue, 6 Jun 2017 23:19:02 +0000 (17:19 -0600)]
rust: build fixes and nom update
Update nom to ~3.0.
Prefix dependencies with ~, which will allow for newer patch
versions only. Minor version updates should get a test before
using.
Remove Cargo.lock from the repo, but still generate as part
of the vendoring process for release builds. This will ensure
that all users of a particular distribution tarball will be
linking against the same Rust dependencies.
Eric Leblond [Mon, 24 Apr 2017 13:00:20 +0000 (15:00 +0200)]
stream-tcp: add option to accept invalid packets
Suricata was inconditionaly dropping packets that are invalid with
respect to the streaming engine. In some corner case like asymetric
trafic capture, this was leading to dropping some legitimate trafic.
The async-oneside option did help but this was not perfect in some
real life case. So this patch introduces an option that allow the
user to tell Suricata not to drop packet that are invalid with
respect to streaming.
Eric Leblond [Wed, 31 May 2017 16:14:29 +0000 (18:14 +0200)]
filestore: avoid open write close sequence
Current file storing approach is using a open file, write data,
close file logic. If this technic is fixing the problem of getting
too much open files in Suricata it is not optimal.
Test on a loop shows that open, write, close on a single file is
two time slower than a single open, loop of write, close.
This patch updates the logic by storing the fd in the File structure.
This is done for a certain number of files. If this amount is exceeded
then the previous logic is used.
This patch also adds two counters. First is the number of
currently open files. The second one is the number of time
the open, write, close sequence has been used due to too much
open files.
In EVE, the entries are:
stats {file_store: {"open_files_max_hit":0,"open_files":5}}