--- /dev/null
+File Extraction
+===============
+
+.. toctree::
+
+ md5
+ filemd5-and-whiteblacklisting-with-md5
+ public-sha1-md5-data-sets
+
+Starting with Suricata version 1.2 it's possible to extract files from HTTP sessions as well as match on file name, extension and "magic".
+
+
+Architecture
+~~~~~~~~~~~~
+
+The file extraction code works on top of the HTTP parser which itself is largely a wrapper for libhtp. The HTTP parser takes care of dechunking and unzipping the request and/or response data if necessary. The HTTP parser runs on top of the stream reassembly engine.
+
+This means that settings in the stream engine, reassembly engine and the HTTP parser all affect the workings of the file extraction.
+
+What files are actually extracted and stored to disk is controlled by the rule language.
+
+
+Settings
+~~~~~~~~
+
+*stream.checksum_validation* controls whether or not the stream engine rejects packets with invalid checksums. A good idea normally, but the network interface performs checksum offloading a lot of packets may seem to be broken. This setting is enabled by default, and can be disabled by setting to "no". Note that the checksum handling can be controlled per interface, see "checksum_checks" in example configuration.
+
+*stream.reassembly.depth* controls how far into a stream reassembly is done. Beyond this value no reassembly will be done. This means that after this value the HTTP session will no longer be tracked. By default a settings of 1 Megabyte is used. 0 sets it to unlimited.
+
+*libhtp.default-config.request-body-limit* / *libhtp.server-config.<config>.request-body-limit* controls how much of the HTTP request body is tracked for inspection by the http_client_body keyword, but also used to limit file inspection. A value of 0 means unlimited.
+
+*libhtp.default-config.response-body-limit* / *libhtp.server-config.<config>.response-body-limit* is like the request body limit, only it applies to the HTTP response body.
+
+NIC offloading
+~~~~~~~~~~~~~~
+
+NIC offloading should be disabled:
+
+::
+
+
+ apt-get install ethtool
+
+
+::
+
+
+ ethtool -k eth3
+
+ Offload parameters for eth3:
+ rx-checksumming: off
+ tx-checksumming: off
+ scatter-gather: off
+ tcp-segmentation-offload: off
+ udp-fragmentation-offload: off
+ generic-segmentation-offload: off
+ generic-receive-offload: off
+ large-receive-offload: off
+ rx-vlan-offload: off
+ tx-vlan-offload: off
+
+Everything should be OFF. If it is not here is how you can disable it:
+
+
+::
+
+
+ ethtool -K eth3 tso off
+ ethtool -K eth3 gro off
+ ethtool -K eth3 lro off
+ ethtool -K eth3 gso off
+ ethtool -K eth3 rx off
+ ethtool -K eth3 tx off
+ ethtool -K eth3 sg off
+ ethtool -K eth3 rxvlan off
+ ethtool -K eth3 txvlan off
+
+NOTICE the difference between small k and a BIG K !!
+Please make sure you choose the appropriate interface name (eth0,eth1,eth5...)
+
+
+Output
+~~~~~~
+
+For file extraction two separate output modules were created: "file-log" and "file-store". They need to be enabled in the [[suricata.yaml]]. For "file-store", the "files" drop dir must be configured.
+
+
+::
+
+
+ - file-store:
+ enabled: yes # set to yes to enable
+ log-dir: files # directory to store the files
+ force-magic: no # force logging magic on all stored files
+ force-md5: no # force logging of md5 checksums
+ waldo: file.waldo # waldo file to store the file_id across runs
+
+Each file that is stored with have a name "file.<id>". The id will be reset and files will be overwritten unless the waldo option is used.
+
+
+::
+
+
+ - file-log:
+ enabled: yes
+ filename: files-json.log
+ append: yes
+ #filetype: regular # 'regular', 'unix_stream' or 'unix_dgram'
+ force-magic: no # force logging magic on all logged files
+ force-md5: no # force logging of md5 checksums
+
+
+Rules
+~~~~~
+
+Without rules in place no extraction will happen. The simplest rule would be:
+
+
+::
+
+
+ alert http any any -> any any (msg:"FILE store all"; filestore; sid:1; rev:1;)
+
+This will simply store all files to disk.
+
+Want to store all files with a pdf extension?
+
+
+::
+
+
+ alert http any any -> any any (msg:"FILE PDF file claimed"; fileext:"pdf"; filestore; sid:2; rev:1;)
+
+Or rather all actual pdf files?
+
+
+::
+
+
+ alert http any any -> any any (msg:"FILE pdf detected"; filemagic:"PDF document"; filestore; sid:3; rev:1;)
+
+Bundled with the Suricata download is a file with more example rules. In the archive, go to the rules/ directory and check the files.rules file.
+
+MD5
+~~~
+
+Suricata can calculate MD5 checksums of files on the fly and log them. See :doc:`md5` for an explanation on how to enable this.
--- /dev/null
+Filemd5 and white or black listing with MD5 hashes
+==================================================
+
+
+
+This guide will show you how to set up a white/black MD5 listing using latest Suricata on Ubuntu LTS 12.04 64 bit
+and a file containing 87 million MD5s - white list.
+
+
+In this set up we had/did the following :
+
+1. A Suricata installation with :doc:`md5` enabled.
+2. We start Suricata with about 4.5K rules and the MD5 white list file containing 87 million MD5 entries.
+3. We have a 9.5Gb of traffic.
+4. We have the following set up:
+
+
+::
+
+
+ CPU: One Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz (16 cores counting Hyperthreading)
+ Memory: 32Gb
+ capture NIC: Intel 82599EB 10-Gigabit SFI/SFP+
+
+
+
+You need to get the white list file containing MD5s from here - http://www.nsrl.nist.gov/Downloads.htm
+This is an official database containing SHA-1 and MD5s for files that are " traceable ".
+For example after you download and unzip (I used the Combo DVD link) you would get a file like so:
+
+
+::
+
+
+
+ "SHA-1","MD5","CRC32","FileName","FileSize","ProductCode","OpSystemCode","SpecialCode"
+ "000000206738748EDD92C4E3D2E823896700F849","392126E756571EBF112CB1C1CDEDF926","EBD105A0","I05002T2.PFB",98865,3095,"WIN",""
+ "0000004DA6391F7F5D2F7FCCF36CEBDA60C6EA02","0E53C14A3E48D94FF596A2824307B492","AA6A7B16","00br2026.gif",2226,228,"WIN",""
+ "000000A9E47BD385A0A3685AA12C2DB6FD727A20","176308F27DD52890F013A3FD80F92E51","D749B562","femvo523.wav",42748,4887,"MacOSX",""
+ "00000142988AFA836117B1B572FAE4713F200567","9B3702B0E788C6D62996392FE3C9786A","05E566DF","J0180794.JPG",32768,16848,"358",""
+ "00000142988AFA836117B1B572FAE4713F200567","9B3702B0E788C6D62996392FE3C9786A","05E566DF","J0180794.JPG",32768,18266,"358",""
+ "00000142988AFA836117B1B572FAE4713F200567","9B3702B0E788C6D62996392FE3C9786A","05E566DF","J0180794.JPG",32768,20316,"358",""
+ "00000142988AFA836117B1B572FAE4713F200567","9B3702B0E788C6D62996392FE3C9786A","05E566DF","J0180794.JPG",32768,20401,"358",""
+ "00000142988AFA836117B1B572FAE4713F200567","9B3702B0E788C6D62996392FE3C9786A","05E566DF","J0180794.JPG",32768,2322,"WIN",""
+ "00000142988AFA836117B1B572FAE4713F200567","9B3702B0E788C6D62996392FE3C9786A","05E566DF","J0180794.JPG",32768,23822,"358",""
+
+
+The file contains 87 million entries/lines, each line having
+"SHA-1","MD5","CRC32","FileName","FileSize","ProductCode","OpSystemCode","SpecialCode"
+of a traceable file.
+
+You can use the Linux "cut" utility to make this file into only a MD5 file - have only the MD5 check sums (second row) - which is what we need.
+Here is what you can do:
+
+Download the ComboDVD - http://www.nsrl.nist.gov/RDS/rds_2.47/RDS_247.iso
+
+unzip/tar, then
+
+::
+
+
+ cat NSRFile.txt | more
+
+would be able to see the beginning of the file and get an idea of what does it contain.(shown above)
+**NOTE – this is a 10GB file, make sure you have the space!**
+
+the MD5 column ... should start with "MD5"
+
+
+::
+
+
+ sudo cut -d'"' -f4 NSRFile.txt >> MD5.txt
+
+and you have only MD5s ...but still with "MD5" on top
+
+
+::
+
+
+ sed -i '1d' MD5.txt
+
+wait for a while
+check the file
+
+
+::
+
+
+ wc -l MD5.txt
+
+wait for about 2-3 min
+
+should be about 87Mil something md5s , one per line
+**NOTE:**
+You can also import the file into a MySQL database, if you would like:
+
+
+::
+
+
+ mysql> create user 'filejson'@'localhost' IDENTIFIED BY 'PASSWORD123';
+ Query OK, 0 rows affected (0.00 sec)
+
+ mysql>create database filejsondb;
+ mysql> grant all privileges on filejsondb.* to 'filejson'@'localhost' with grant option;
+ mysql> flush privileges;
+ mysql> use filejsondb;
+
+Then:
+
+::
+
+
+ CREATE TABLE MD5(md5_whitelist VARCHAR(32));
+
+The trick here for the import is that **the table name and the file name MUST be the same** - aka MySQL table "MD5" and the file is called MD5.txt!
+
+::
+
+
+ sudo mysqlimport -u root -p --local filejsondb MD5.txt
+
+Where "filejsondb" is the name of the database.
+
+
+
+So , here is how we did it ....
+
+
+
+::
+
+
+ pevman@suricata:~$ ls -lh /etc/suricata/et-config/MD5_NSRFile.txt
+ -rw-r--r-- 1 root root 2.7G Aug 29 00:35 /etc/suricata/et-config/MD5_NSRFile.txt
+
+ pevman@suricata:~$ wc -l /etc/suricata/et-config/MD5_NSRFile.txt
+ 87345542 /etc/suricata/et-config/MD5_NSRFile.txt
+ pevman@suricata:~$
+
+
+
+
+
+
+::
+
+
+ pevman@suricata:~$ sudo tcpstat -i eth3
+ Time:1346241952 n=6664547 avg=898.01 stddev=757.68 bps=9575769952.00
+ Time:1346241957 n=6670750 avg=897.22 stddev=754.07 bps=9576254160.00
+ Time:1346241962 n=6626520 avg=903.62 stddev=747.26 bps=9580577822.40
+ Time:1346241967 n=6685956 avg=895.15 stddev=749.34 bps=9575883715.20
+ Time:1346241972 n=6712481 avg=891.53 stddev=747.34 bps=9575027134.40
+ Time:1346241977 n=6696189 avg=893.82 stddev=746.62 bps=9576294273.60
+ Time:1346241982 n=6681826 avg=895.75 stddev=749.67 bps=9576376033.60
+
+
+
+
+
+::
+
+
+ pevman@suricata:~$ suricata --build-info
+ [2746] 29/8/2012 -- 15:07:25 - (suricata.c:539) <Info> (SCPrintBuildInfo) -- This is Suricata version 1.4dev (rev 75af345)
+ [2746] 29/8/2012 -- 15:07:25 - (suricata.c:612) <Info> (SCPrintBuildInfo) -- Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK HAVE_HTP_TX_GET_RESPONSE_HEADERS_RAW HAVE_NSS
+ [2746] 29/8/2012 -- 15:07:25 - (suricata.c:626) <Info> (SCPrintBuildInfo) -- 64-bits, Little-endian architecture
+ [2746] 29/8/2012 -- 15:07:25 - (suricata.c:628) <Info> (SCPrintBuildInfo) -- GCC version 4.6.3, C version 199901
+ [2746] 29/8/2012 -- 15:07:25 - (suricata.c:634) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_1
+ [2746] 29/8/2012 -- 15:07:25 - (suricata.c:637) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_2
+ [2746] 29/8/2012 -- 15:07:25 - (suricata.c:640) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4
+ [2746] 29/8/2012 -- 15:07:25 - (suricata.c:643) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8
+ [2746] 29/8/2012 -- 15:07:25 - (suricata.c:646) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_16
+ [2746] 29/8/2012 -- 15:07:25 - (suricata.c:650) <Info> (SCPrintBuildInfo) -- compiled with -fstack-protector
+ [2746] 29/8/2012 -- 15:07:25 - (suricata.c:656) <Info> (SCPrintBuildInfo) -- compiled with _FORTIFY_SOURCE=2
+
+During Suricata start up :
+...
+
+::
+
+
+ [3071] 29/8/2012 -- 15:23:45 - (detect.c:670) <Info> (SigLoadSignatures) -- Loading rule file: /var/data/peter/md5test.rules
+ [3071] 29/8/2012 -- 15:23:45 - (detect-filemd5.c:105) <Error> (Md5ReadString) -- [ERRCODE: SC_ERR_INVALID_MD5(214)] - md5 string not 32 bytes
+ [3071] 29/8/2012 -- 15:24:25 - (detect-filemd5.c:277) <Info> (DetectFileMd5Parse) -- MD5 hash size 1399625840 bytes, negated match
+ [3071] 29/8/2012 -- 15:24:25 - (detect.c:701) <Info> (SigLoadSignatures) -- 5 rule files processed. 4641 rules succesfully loaded, 0 rules failed
+
+
+You will get the
+
+::
+
+ [ERRCODE: SC_ERR_INVALID_MD5(214)] - md5 string not 32 bytes
+
+if a line of the file is not containing a proper MD5 hash.
+
+For example above we get the err message (which is more of a warning) because in the file , the first line was containing:
+"SHA-1","MD5","CRC32","FileName","FileSize","ProductCode","OpSystemCode","SpecialCode"
+
+However , nonetheless Suricata will continue loading the rest of the MD5 hashes from the file. REMEMBER - you would need put only the MD5 hashes in the file.
+
+
+Then we just created the following test rules:
+
+::
+
+
+ root@suricata:/etc/suricata/peter-yaml# cat /var/data/peter/md5test.rules
+ alert http any any -> any any (msg:"FILE MD5 Check PDF against a white list"; filemagic:"pdf"; filemd5:!MD5_NSRFile.txt; sid:9966699; rev:1;)
+ alert http any any -> any any (msg:"FILE MD5 Check EXE against a white list"; filemagic:"exe"; filemd5:!MD5_NSRFile.txt; sid:9977799; rev:2;)
+
+Make sure the **MD5_NSRFile.txt** file (containing the MD5 hashes) is in your "rules directory" (where you load the rules from).
+
+
+Basically the two rules above are telling Suricata to do the following:
+1. If you see a PDF document that has a MD5 hash NOT in the MD5_NSRFile.txt - generate an alert
+2. If you see an EXE file that has a MD5 hash NOT in the MD5_NSRFile.txt - generate an alert
+
+all that on the fly, while inspecting traffic.
+
+
+Then all that is left is to start Suricata:
+
+::
+
+
+ sudo /usr/local/bin/suricata -c /etc/suricata/peter-yaml/suricata-af-packet-mmap.yaml -s /var/data/peter/md5test.rules --af-packet=eth3
+
+and we get the alerts:
+
+
+::
+
+
+ 08/29/2012-15:38:43.165038 [**] [1:9977799:2] FILE MD5 Check EXE against a white list [**] [Classification: (null)] [Priority: 3] {TCP} y.y.y.y:80 -> x.x.x.x:23836
+ 08/29/2012-15:39:32.551950 [**] [1:9977799:2] FILE MD5 Check EXE against a white list [**] [Classification: (null)] [Priority: 3] {TCP} y.y.y.y:2091 -> x.x.x.x:80
+
+
+
+That's it.
+
+You can reverse and use the above rules with a "blacklisting" of MD5 hashes, for example:
+
+
+::
+
+
+ alert http any any -> any any (msg:"FILE MD5 Check PDF against a black list"; filemagic:"pdf"; filemd5:BlackMD5s.txt; sid:9966699; rev:1;)
+
+You can also use the filestore keyword to store the file on disk and do further analysis on the particular file - or blend it in with other :doc:`../rules/file-keywords`.
+
+Peter Manev
--- /dev/null
+Storing MD5s checksums
+======================
+
+In this particular example we are using: Ubuntu 14.04 LTS
+
+Also - we are using the latest git master (git installation)
+
+Make sure you have libnss and libnspr installed
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+
+::
+
+
+ root@LTS-64-1:~/Work/tmp/oisf# dpkg -l |grep libnss
+ ii libnss-mdns:amd64 0.10-6 amd64 NSS module for Multicast DNS name resolution
+ ii libnss3:amd64 2:3.17.4-0ubuntu0.14.04.1 amd64 Network Security Service libraries
+ ii libnss3-1d:amd64 2:3.17.4-0ubuntu0.14.04.1 amd64 Network Security Service libraries - transitional package
+ ii libnss3-dev:amd64 2:3.17.4-0ubuntu0.14.04.1 amd64 Development files for the Network Security Service libraries
+ ii libnss3-nssdb 2:3.17.4-0ubuntu0.14.04.1 all Network Security Security libraries - shared databases
+ ii libnss3-tools 2:3.17.4-0ubuntu0.14.04.1 amd64 Network Security Service tools
+
+
+
+::
+
+
+ root@LTS-64-1:~/Work/tmp/oisf# dpkg -l |grep libnspr
+ ii libnspr4:amd64 2:4.10.7-0ubuntu0.14.04.1 amd64 NetScape Portable Runtime Library
+ ii libnspr4-dev 2:4.10.7-0ubuntu0.14.04.1 amd64 Development files for the NetScape Portable Runtime library
+
+If not install them:
+
+::
+
+
+ apt-get install libnss3-dev libnspr4-dev
+
+**Note:** Fedora users need to install the following:
+
+::
+
+
+ nss-util
+ nss-util-devel
+ nss-devel
+ nspr-devel
+ nspr
+
+Get the Suricata code
+~~~~~~~~~~~~~~~~~~~~~
+
+Execute:
+
+::
+
+
+ git clone git://phalanx.openinfosecfoundation.org/oisf.git && cd oisf
+ git clone https://github.com/OISF/libhtp.git -b 0.5.x
+
+Building Suricata
+~~~~~~~~~~~~~~~~~~
+
+You have to compile/install suri like this in order to enable MD5s:
+
+::
+
+
+ ./autogen.sh
+ ./configure --with-libnss-libraries=/usr/lib --with-libnss-includes=/usr/include/nss/ --with-libnspr-libraries=/usr/lib --with-libnspr-includes=/usr/include/nspr
+ make clean
+ make
+ sudo make install
+
+For Fedora 64-bit users, the configure line will look like
+
+
+::
+
+
+ ./configure --with-libnss-libraries=/usr/lib64 --with-libnss-includes=/usr/include/nss3 --with-libnspr-libraries=/usr/lib64 --with-libnspr-includes=/usr/include/nspr4
+
+Output of configure:
+
+
+::
+
+
+ Suricata Configuration:
+ AF_PACKET support: yes
+ PF_RING support: no
+ NFQueue support: no
+ IPFW support: no
+ DAG enabled: no
+ Napatech enabled: no
+
+ libnss support: yes
+ libnspr support: yes
+ Prelude support: no
+ PCRE jit: no
+
+This is what is important to have:
+
+::
+
+
+ libnss support: yes
+ libnspr support: yes
+
+Confirm everything is built correctly:
+
+
+::
+
+
+ # suricata --build-info
+ [10010] 1/5/2012 -- 11:16:23 - (suricata.c:502) <Info> (SCPrintBuildInfo) -- This is Suricata version 1.3dev (rev e6dea5c)
+ [10010] 1/5/2012 -- 11:16:23 - (suricata.c:575) <Info> (SCPrintBuildInfo) -- Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK HAVE_HTP_TX_GET_RESPONSE_HEADERS_RAW HAVE_NSS
+ [10010] 1/5/2012 -- 11:16:23 - (suricata.c:589) <Info> (SCPrintBuildInfo) -- 32-bits, Little-endian architecture
+ [10010] 1/5/2012 -- 11:16:23 - (suricata.c:591) <Info> (SCPrintBuildInfo) -- GCC version 4.4.5, C version 199901
+ [10010] 1/5/2012 -- 11:16:23 - (suricata.c:597) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_1
+ [10010] 1/5/2012 -- 11:16:23 - (suricata.c:600) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_2
+ [10010] 1/5/2012 -- 11:16:23 - (suricata.c:603) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4
+ [10010] 1/5/2012 -- 11:16:23 - (suricata.c:606) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8
+ [10010] 1/5/2012 -- 11:16:23 - (suricata.c:613) <Info> (SCPrintBuildInfo) -- compiled with -fstack-protector
+ [10010] 1/5/2012 -- 11:16:23 - (suricata.c:619) <Info> (SCPrintBuildInfo) -- compiled with _FORTIFY_SOURCE=2
+
+Make sure we have **HAVE_NSS** in the **Features** line.
+
+Configuration
+~~~~~~~~~~~~~
+
+In the suricata yaml:
+
+
+::
+
+
+ - file-store:
+ enabled: yes # set to yes to enable
+ log-dir: files # directory to store the files
+ force-magic: yes # force logging magic on all stored files
+ force-md5: yes # force logging of md5 checksums
+ #waldo: file.waldo # waldo file to store the file_id across runs
+
+Optionally, for JSON output:
+
+
+::
+
+
+ - file-log:
+ enabled: yes
+ filename: files-json.log
+ append: no
+
+Other settings affecting :doc:`file-extraction`
+
+
+::
+
+
+ stream:
+ memcap: 64mb
+ checksum-validation: yes # reject wrong csums
+ inline: no # no inline mode
+ reassembly:
+ memcap: 32mb
+ depth: 0 # reassemble all of a stream
+ toserver-chunk-size: 2560
+ toclient-chunk-size: 2560
+
+Make sure we have *depth: 0* so all files can be tracked fully.
+
+
+::
+
+
+ libhtp:
+ default-config:
+ personality: IDS
+ # Can be specified in kb, mb, gb. Just a number indicates
+ # it's in bytes.
+ request-body-limit: 0
+ response-body-limit: 0
+
+Make sure we have *request-body-limit: 0* and *response-body-limit: 0*
+
+Testing
+~~~~~~~
+
+For the purpose of testing we use this rule only in a file.rules (a test/example file):
+
+
+::
+
+
+ alert http any any -> any any (msg:"FILE store all"; filestore; sid:1; rev:1;)
+
+This rule above will save all the file data for files that are opened/downloaded through HTTP
+
+Start Suricta (-S option loads ONLY the specified rule file, with disregard if any other rules that are enabled in suricata.yaml):
+
+
+::
+
+
+ suricata -c /etc/suricata/suricata.yaml -S file.rules -i eth0
+
+
+I tried that link (Cisco Prod Brochure PDF, just googled "Cisco PDF"):
+
+* http://www.cisco.com/c/en/us/products/routers/3800-series-integrated-services-routers-isr/index.html
+
+and in file directory (/var/log/suricata/files) I got the meta data:
+
+
+
+::
+
+
+ TIME: 05/01/2012-11:09:52.425751
+ SRC IP: 2.23.144.170
+ DST IP: 192.168.1.91
+ PROTO: 6
+ SRC PORT: 80
+ DST PORT: 51598
+ HTTP URI: /en/US/prod/collateral/routers/ps5855/prod_brochure0900aecd8019dc1f.pdf
+ HTTP HOST: www.cisco.com
+ HTTP REFERER: http://www.cisco.com/c/en/us/products/routers/3800-series-integrated-services-routers-isr/index.html
+ FILENAME: /en/US/prod/collateral/routers/ps5855/prod_brochure0900aecd8019dc1f.pdf
+ MAGIC: PDF document, version 1.6
+ STATE: CLOSED
+ MD5: 59eba188e52467adc11bf2442ee5bf57
+ SIZE: 9485123
+
+and in files-json.log (or eve.json) :
+
+
+::
+
+
+ { "id": 1, "timestamp": "05\/01\/2012-11:10:27.693583", "ipver": 4, "srcip": "2.23.144.170", "dstip": "192.168.1.91", "protocol": 6, "sp": 80, "dp": 51598, "http_uri": "\/en\/US\/prod\/collateral\/routers\/ps5855\/prod_brochure0900aecd8019dc1f.pdf", "http_host": "www.cisco.com", "http_referer": "http:\/\/www.google.com\/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CDAQFjAA&url=http%3A%2F%2Fwww.cisco.com%2Fen%2FUS%2Fprod%2Fcollateral%2Frouters%2Fps5855%2Fprod_brochure0900aecd8019dc1f.pdf&ei=OqyfT9eoJubi4QTyiamhAw&usg=AFQjCNGdjDBpBDfQv2r3VogSH41V6T5x9Q", "filename": "\/en\/US\/prod\/collateral\/routers\/ps5855\/prod_brochure0900aecd8019dc1f.pdf", "magic": "PDF document, version 1.6", "state": "CLOSED", "md5": "59eba188e52467adc11bf2442ee5bf57", "stored": true, "size": 9485123 }
+ { "id": 12, "timestamp": "05\/01\/2012-11:12:57.421420", "ipver": 4, "srcip": "2.23.144.170", "dstip": "192.168.1.91", "protocol": 6, "sp": 80, "dp": 51598, "http_uri": "\/en\/US\/prod\/collateral\/routers\/ps5855\/prod_brochure0900aecd8019dc1f.pdf", "http_host": "www.cisco.com", "http_referer": "http:\/\/www.google.com\/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CDAQFjAA&url=http%3A%2F%2Fwww.cisco.com%2Fen%2FUS%2Fprod%2Fcollateral%2Frouters%2Fps5855%2Fprod_brochure0900aecd8019dc1f.pdf&ei=OqyfT9eoJubi4QTyiamhAw&usg=AFQjCNGdjDBpBDfQv2r3VogSH41V6T5x9Q", "filename": "\/en\/US\/prod\/collateral\/routers\/ps5855\/prod_brochure0900aecd8019dc1f.pdf", "magic": "PDF document, version 1.6", "state": "CLOSED", "md5": "59eba188e52467adc11bf2442ee5bf57", "stored": true, "size": 9485123 }
+
+
+Log all MD5s without any rules
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you would like to log MD5s for everything and anything that passes through the traffic that you are inspecting with Suricata, but not log the files themselves, all you have to do is disable file-store and enable only the JSON output with forced MD5s - in suricata.yaml like so:
+
+
+::
+
+
+ - file-store:
+ enabled: no # set to yes to enable
+ log-dir: files # directory to store the files
+ force-magic: yes # force logging magic on all stored files
+ force-md5: yes # force logging of md5 checksums
+ #waldo: file.waldo # waldo file to store the file_id across runs
+
+ - file-log:
+ enabled: yes
+ filename: files-json.log
+ append: no
+ #filetype: regular # 'regular', 'unix_stream' or 'unix_dgram'
+ force-magic: yes # force logging magic on all logged files
+ force-md5: yes # force logging of md5 checksums
+
+This is in short what is needed to have MD5s logged.
--- /dev/null
+Public SHA1 MD5 data sets
+=========================
+
+
+
+
+National Software Reference Library - http://www.nsrl.nist.gov/Downloads.htm
reputation/index.rst
initscripts
setting-up-ipsinline-for-linux
+ file-extraction/file-extraction.rst
public-data-sets
capture-hardware/index.rst
projects / thirdparty / suricata.git / commitdiff
