From: Hui Cao (huica) Date: Tue, 22 Nov 2016 21:40:14 +0000 (-0500) Subject: Merge pull request #718 in SNORT/snort3 from sdf_documenation to master X-Git-Tag: 3.0.0-233~176 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=7799c0f81c8fb1966231626086dec1e2325a2d5f;p=thirdparty%2Fsnort3.git Merge pull request #718 in SNORT/snort3 from sdf_documenation to master Squashed commit of the following: commit 3cfa955237b9a1e87ca3cd08a9f8c8d2d7c81a8d Author: Victor Roemer Date: Tue Nov 15 14:50:01 2016 -0500 sd_pattern user documentation --- diff --git a/doc/features.txt b/doc/features.txt index 360cdf1e9..a6d90691a 100644 --- a/doc/features.txt +++ b/doc/features.txt @@ -6,3 +6,6 @@ include::file_processing.txt[] include::perf_monitor.txt[] +=== Sensitive Data Filtering + +include::sensitive_data.txt[] diff --git a/doc/sensitive_data.txt b/doc/sensitive_data.txt new file mode 100644 index 000000000..016147c1e --- /dev/null +++ b/doc/sensitive_data.txt @@ -0,0 +1,121 @@ +=== Overview + +The `sd_pattern` IPS option provides detection and filtering of Personally +Identifiable Information (PII). This information includes credit card +numbers, U.S. Social Security numbers, and email addresses. A rich regular +expression syntax is available for defining your own PII. + +=== Dependencies + +==== Hyperscan + +The `sd_pattern` rule option is powered by the open source Hyperscan +library from Intel. It provides a regex grammar which is mostly PCRE +compatible. To learn more about Hyperscan see +http://01org.github.io/hyperscan/dev-reference/ + +=== Syntax + +Snort provides `sd_pattern` as IPS rule option with no additional inspector +overhead. The Rule option takes the following syntax. + + sd_pattern: ""[, threshold ]; + +===== Pattern + +Pattern is the most important and is the only required parameter to +`sd_pattern`. It supports 3 built in patterns which are configured by name: +"credit_card", "us_social" and "us_social_nodashes", as well as user +defined regular expressions of the Hyperscan dialect (see +http://01org.github.io/hyperscan/dev-reference/compilation.html#pattern-support). + + sd_pattern:"credit_card"; + +When configured, Snort will replace the pattern 'credit_card' with the built in +pattern. In addition to pattern matching, Snort will validate that the matched +digits will pass the Luhn-check algorithm. Currently the only pattern that +performs extra verification. + + sd_pattern:"us_social"; + sd_pattern:"us_social_nodashes"; + +These special patterns will also be replaced with a built in pattern. +Naturally, "us_social" is a pattern of 9 digits separated by `-`'s in the +canonical form. + + sd_pattern:"\b\w+@ourdomain\.com\b" + +This is a user defined pattern which matches what is most likely email +addresses for the site "ourdomain.com". The pattern is a PCRE compatible +regex, '\b' matches a word boundary (whitespace, end of line, non-word +characters) and '\w+' matches one or more word characters. '\.' matches +a literal '.'. + +The above pattern would match "a@ourdomain.com", "aa@ourdomain.com" but +would not match "1@ourdomain.com", "ab12@ourdomain.com" or "@ourdomain.com". + +Note: This is just an example, this pattern is not suitable to detect many +correctly formatted emails. + +===== Threshold + +Threshold is an optional parameter allowing you to change built in default +value (default value is '1'). The following two instances are identical. +The first will assume the default value of '1' the second declaration +explicitly sets the threshold to '1'. + + sd_pattern:"This rule requires 1 match"; + sd_pattern:"This rule requires 1 match", threshold 1; + +That's pretty easy, but here is one more example anyway. + + sd_pattern:"This is a string literal", threshold 300; + +This example requires 300 matches of the pattern "This is a string literal" +to qualify as a positive match. That is, if the string only occurred 299x +in a packet, you will not see an even. + +==== Obfuscating Credit Cards and Social Security Numbers + +Snort provides discreet logging for the built in patterns "credit_card", +"us_social" and "us_social_nodashes". Enabling `output.obfuscate_pii` makes +Snort obfuscate the suspect packet payload which was matched by the +patterns. This configuration is disabled by default. + + output = + { + obfuscate_pii = true + } + +===== Example + +A complete Snort IPS rule + + alert tcp ( sid:1; msg:"Credit Card"; sd_pattern:"credit_card"; ) + +Logged output when running Snort in "cmg" alert format. + + 02/25-21:19:05.125553 [**] [1:1:0] "Credit Card" [**] [Priority: 0] {TCP} 10.1.2.3:48620 -> 10.9.8.7:8 + 02:01:02:03:04:05 -> 02:09:08:07:06:05 type:0x800 len:0x46 + 10.1.2.3:48620 -> 10.9.8.7:8 TCP TTL:64 TOS:0x0 ID:14 IpLen:20 DgmLen:56 + ***A**** Seq: 0xB2 Ack: 0x2 Win: 0x2000 TcpLen: 20 + - - - raw[16] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 39 32 39 34 ............9294 + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + +==== Caveats + +1. Snort currently requires setting the fast pattern engine to use +"hyperscan" in order for `sd_pattern` ips option to function correctly. + + search_engine = { search_method = 'hyperscan' } + +2. Log obfuscation is only applicable to CMG and Unified2 logging formats. + +3. Log obfuscation doesn't support user defined PII patterns. It is +currently only supported for the built in patterns for Credit Cards and US +Social Security numbers. + +4. Log obfuscation doesn't work with stream rebuilt packet payloads. (This +is a known bug). +