+eBPF and XDP
+============
-Set up
-======
+Introduction
+------------
+
+eBPF stands for extended BPF. This is an extended version of Berkeley Packet Filter available in recent
+Linux kernel versions.
+
+It provides more advanced features with eBPF programs developed in C and able to use structured data shared
+between kernel and userspace.
+
+eBPF is used for three things in Suricata:
+
+- eBPF filter: any BPF like filter can be developed. An example of filter accepting only packet for some VLANs is provided.
+- eBPF load balancing: provide programmable load balancing. A simple ippair load balancing is provided.
+- XDP programs: suricata can load XDP programs. A bypass program is provided.
+
+Bypass can be implemented in eBPF and XDP. The advantage of XDP is that the packets are dropped at the earliest stage
+possible. So performance is better. But bypassed packets don't reach the network so you can't use this on regular
+traffic but only on duplicated/sniffed traffic.
XDP
----
+~~~
XDP provides another Linux native way of optimising Suricata's performance on sniffing high speed networks.
Requirements
------------
-- Newer kernel that supports XDP (4.13.10 in our case)
-- RSS symmetric hashing on the NIC ( Intel 82599ES 10-Gigabit/x520/x540 in our case)
-- In tree kernel drivers NIC drivers
+You will need a kernel that supports XDP and, for real performance improvement, a network
+card that support XDP in the driver.
+
+Suricata XDP code has been tested with 4.13.10 but 4.15 or later is necessary to have all
+features like the CPU redirect map.
+
+If yu are using an Intel netword card, you will need to stay with in tree kernel NIC drivers.
+The out of tree drivers do not contain the XDP support.
+
+Having a network card with support for RSS symmetric hashing is a good point or you will have to
+use the XDP CPU redirect map feature.
Prerequisites
-------------
-This guide has been confirmed on Debian/Ubutnu LTS Linux.
+This guide has been confirmed on Debian/Ubuntu "LTS" Linux.
Disable irqbalance
~~~~~~~~~~~~~~~~~~
Clang
~~~~~
-Make sure you have clang installed on the system.
-::
+Make sure you have clang (>=3.9) installed on the system ::
- apt-get install clang
+ sudo apt-get install clang
+
+The version 3.9 is recommended as it seems some more recent versions of clang
+generate invalid eBPF code that fail to load.
BPF
~~~
-Suricata use libbpf to interact with eBPF and XDP. This library is available
-in the Linux tree. Before 4.16, a patched libbpf library is also needed::
+Suricata uses libbpf to interact with eBPF and XDP. This library is available
+in the Linux tree. Before Linux 4.16, a patched libbpf library is also needed::
- git clone -b libbpf-xdp https://github.com/regit/linux.git
+ git clone -b libbpf-v14 https://github.com/regit/linux.git
If you have a recent enough kernel, you can skip this part.
cd linux/tools/lib/bpf/
make && sudo make install
- sudo mkdir -p /usr/local/include/bpf/
- sudo cp *bpf.h /usr/local/include/bpf/
+ sudo make install_headers
sudo ldconfig
Compile and install Suricata
----------------------------
-::
+If ever you don't have the source ::
git clone https://github.com/OISF/suricata.git
cd suricata && git clone https://github.com/OISF/libhtp.git -b 0.5.x
./autogen.sh
+Then you need to add the ebpf flags to configure ::
+
CC=clang ./configure --prefix=/usr/ --sysconfdir=/etc/ --localstatedir=/var/ \
--enable-ebpf --enable-ebpf-build
sudo ldconfig
sudo mkdir /etc/suricata/ebpf/
+Setup bypass
+------------
+
+If you plan to use eBPF or XDP for a kernel/hardware level bypass, you need to do
+the following:
+
+First, enable `bypass` in the `stream` section ::
+
+ stream:
+ bypass: true
+
+If you want, you can also bypass encrypted flow by setting `no-reassemble` to `yes`
+in the app-layer tls section ::
+
+ app-layer:
+ protocols:
+ tls:
+ enabled: yes
+ detection-ports:
+ dp: 443
+
+ # Completely stop processing TLS/SSL session after the handshake
+ # completed. If bypass is enabled this will also trigger flow
+ # bypass. If disabled (the default), TLS/SSL session is still
+ # tracked for Heartbleed and other anomalies.
+ no-reassemble: yes
+
+
+Setup eBPF filter
+-----------------
+
+Copy the resulting ebpf fiter as needed ::
+
+ cp src/ebpf/vlan_filter.bpf /etc/suricata/ebpf/
+
+Then setup the `ebpf-filter-file` variable in af-packet section ::
+
+ - interface: eth3
+ threads: 16
+ cluster-id: 97
+ cluster-type: cluster_flow # choose any type suitable
+ defrag: yes
+ # eBPF file containing a 'loadbalancer' function that will be inserted into the
+ # kernel and used as load balancing function
+ ebpf-filter-file: /etc/suricata/ebpf/vlan_filter.bpf
+ use-mmap: yes
+ ring-size: 200000
+
+You can then run suricata normally ::
+
+ /usr/bin/suricata --pidfile /var/run/suricata.pid --af-packet=eth3 -vvv
+
+You can also use eBPF bypass. To do that load the `bypass_filter.bpf` file and
+update af-packet configuration to set bypass to yes ::
+
+ - interface: eth3
+ threads: 16
+ cluster-id: 97
+ cluster-type: cluster_qm # symmetric hashing is a must!
+ defrag: yes
+ # eBPF file containing a 'loadbalancer' function that will be inserted into the
+ # kernel and used as load balancing function
+ #ebpf-lb-file: /etc/suricata/ebpf/lb.bpf
+ # eBPF file containing a 'filter' function that will be inserted into the
+ # kernel and used as packet filter function
+ # eBPF file containing a 'xdp' function that will be inserted into the
+ # kernel and used as XDP packet filter function
+ ebpf-filter-file: /etc/suricata/ebpf/bypass_filter.bpf
+ bypass: yes
+ use-mmap: yes
+ ring-size: 200000
+
+
+Setup eBPF load balancing
+-------------------------
+
+Copy the resulting ebpf fiter as needed ::
+
+ cp src/ebpf/lb.bpf /etc/suricata/
+
+We will use ``cluster_ebpf`` in the interface section of af-packet ::
+
+ - interface: eth3
+ threads: 16
+ cluster-id: 97
+ cluster-type: cluster_ebpf
+ defrag: yes
+ # eBPF file containing a 'loadbalancer' function that will be inserted into the
+ # kernel and used as load balancing function
+ ebpf-lb-file: /etc/suricata/ebpf/lb.bpf
+ use-mmap: yes
+ ring-size: 200000
+
+Setup XDP
+---------
+
Copy the resulting xdp fiter as needed::
cp src/ebpf/xdp_filter.bpf /etc/suricata/ebpf/
- interface: eth3
threads: 16
cluster-id: 97
- cluster-type: cluster_qm # symmetric hashing is a must!
+ cluster-type: cluster_qm # symmetric hashing is a must!
defrag: yes
# eBPF file containing a 'loadbalancer' function that will be inserted into the
# kernel and used as load balancing function
use-mmap: yes
ring-size: 200000
-Also enable "bypass" in the "stream" section ::
-
- stream:
- bypass: true
-
-If you with you can also bypass encrypted flow by setting no-reassemble to yes
-in the app-layer tls section ::
-
- app-layer:
- protocols:
- tls:
- enabled: yes
- detection-ports:
- dp: 443
-
- # Completely stop processing TLS/SSL session after the handshake
- # completed. If bypass is enabled this will also trigger flow
- # bypass. If disabled (the default), TLS/SSL session is still
- # tracked for Heartbleed and other anomalies.
- no-reassemble: yes
-
Setup symmetric hashing on the NIC
-----------------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Intel network card don't support symmetric hashing but it is possible to emulate
+it by using a specific hashing function.
Follow these instructions closely for desired result::
ifconfig eth3 down
-Use in tree kernel drivers, XDP support is not available in out of source Intel drivers.
-
-*NOTE:* In this case the kernel and sources used is 4.13.10
-
-Reload the module with updated parameters ::
-
- cd /lib/modules/4.13.10-amd64/kernel/drivers/net/ethernet/intel/ixgbe
- rmmod ixgbe && insmod ixgbe.ko MQ=1,1 RSS=0,0 InterruptThrottleRate=12500,12500 LRO=0,0 vxlan_rx=0,0
+Use in tree kernel drivers: XDP support is not available in Intel drivers available on Intel website.
Enable symmetric hashing::
ethtool -x eth3
ethtool -n eth3
-In the above set up you are free to use any recent ``set_irq_affinity`` script. It is available in any Intel x520/710 NIC sources driver download.
+In the above setup you are free to use any recent ``set_irq_affinity`` script. It is available in any Intel x520/710 NIC sources driver download.
**NOTE:**
We use a special low entropy key for the symmetric hashing. `More info about the research for symmetric hashing set up <http://www.ndsl.kaist.edu/~kyoungsoo/papers/TR-symRSS.pdf>`_
-Disable an NIC offloading
--------------------------
+Disable any NIC offloading
+~~~~~~~~~~~~~~~~~~~~~~~~~~
-Run the following to disable offloading ::
+Run the following command to disable offloading ::
for i in rx tx tso ufo gso gro lro tx nocache copy sg txvlan rxvlan; do
/sbin/ethtool -K eth3 $i off 2>&1 > /dev/null;
done
Balance as much as you can
---------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~~~
Try to use the network's card balancing as much as possible ::
done
Start Suricata with XDP
------------------------
+~~~~~~~~~~~~~~~~~~~~~~~
You can now start Suricata with XDP bypass activated ::
# Requires at least Linux 3.14.
# * cluster_rollover: kernel rotates between sockets filling each socket before moving
# to the next. Requires at least Linux 3.10.
- # * cluster_ebpf: kernel rotates between sockets following an extended BPF function loaded
- # from a file. User needs to use ebpf-lb-file option to set the path to the file
- # to load. Requires at least Linux 4.2.
+ # * cluster_ebpf: eBPF file load balancing. See doc/userguide/capture/ebpf-xdt.rst for
+ # more info.
# Recommended modes are cluster_flow on most boxes and cluster_cpu or cluster_qm on system
# with capture card using RSS (require cpu affinity tuning and system irq tuning)
cluster-type: cluster_flow
# In some fragmentation case, the hash can not be computed. If "defrag" is set
# to yes, the kernel will do the needed defragmentation before sending the packets.
defrag: yes
- # eBPF file containing a 'loadbalancer' function that will be inserted into the
- # kernel and used as load balancing function
- #ebpf-lb-file: @e_sysconfdir@/ebpf/lb.bpf
- # eBPF file containing a 'filter' function that will be inserted into the
- # kernel and used as packet filter function
- #ebpf-filter-file: @e_sysconfdir@/ebpf/filter.bpf
- # eBPF file containing a 'xdp' function that will be inserted into the
- # kernel and used as XDP packet filter function
- #ebpf-filter-file: @e_sysconfdir@filter.bpf
- # Xdp mode, "soft" for skb based version, "driver" for network card based
- # and "hw" for card supporting eBPF.
- #xdp-mode: driver
- #xdp-filter-file: @e_sysconfdir@xdp_filter.bpf
- # if the ebpf filter implements a bypass function, you can set 'bypass' to
- # yes and benefit from these feature
- #bypass: yes
# After Linux kernel 3.10 it is possible to activate the rollover option: if a socket is
# full then kernel will send the packet on the next socket with room available. This option
# can minimize packet drop and increase the treated bandwidth on single intensive flow.
# will not be copied.
#copy-mode: ips
#copy-iface: eth1
+ # For eBPF and XDP setup including bypass, filter and load balancing, please
+ # see doc/userguide/capture/ebpf-xdt.rst for more info.
# Put default values here. These will be used for an interface that is not
# in the list above.
projects / thirdparty / suricata.git / commitdiff
