]> git.ipfire.org Git - thirdparty/vectorscan.git/log
thirdparty/vectorscan.git
7 years agofdr_compile: faster scoring code
Justin Viiret [Wed, 9 Aug 2017 04:04:28 +0000 (14:04 +1000)] 
fdr_compile: faster scoring code

7 years agoinsertion_ordered_{map,set}: add new containers
Justin Viiret [Fri, 4 Aug 2017 03:23:07 +0000 (13:23 +1000)] 
insertion_ordered_{map,set}: add new containers

These are associative map/set structures that are iterable in insertion
order.

7 years agoviolet: don't bother swapping holders if unable to trim graph
Alex Coyte [Tue, 8 Aug 2017 01:24:52 +0000 (11:24 +1000)] 
violet: don't bother swapping holders if unable to trim graph

7 years agoviolet: maintain a reference to all the known implementable graphs
Alex Coyte [Tue, 8 Aug 2017 00:13:46 +0000 (10:13 +1000)] 
violet: maintain a reference to all the known implementable graphs

7 years agoviolet: do not remove more states from holders if dfa has been built
Alex Coyte [Mon, 7 Aug 2017 06:41:13 +0000 (16:41 +1000)] 
violet: do not remove more states from holders if dfa has been built

7 years agoUE-3147: ensure the same squash behavior for literals shared between different litera...
Wang, Xiang W [Mon, 7 Aug 2017 14:02:53 +0000 (10:02 -0400)] 
UE-3147: ensure the same squash behavior for literals shared between different literal matchers

7 years agogroupByFragment: make fewer string copies
Justin Viiret [Fri, 4 Aug 2017 00:06:20 +0000 (10:06 +1000)] 
groupByFragment: make fewer string copies

7 years agorose_build_exclusive: clean up use of vertex indices
Justin Viiret [Thu, 3 Aug 2017 05:52:25 +0000 (15:52 +1000)] 
rose_build_exclusive: clean up use of vertex indices

7 years agolimex_compile: reduce state id lookups
Justin Viiret [Fri, 14 Jul 2017 03:01:00 +0000 (13:01 +1000)] 
limex_compile: reduce state id lookups

7 years agolimex_compile: turn tugs into a bitset
Justin Viiret [Fri, 14 Jul 2017 02:54:03 +0000 (12:54 +1000)] 
limex_compile: turn tugs into a bitset

7 years agorose_build_lookaround: use vector in trimLiterals
Justin Viiret [Thu, 3 Aug 2017 01:33:42 +0000 (11:33 +1000)] 
rose_build_lookaround: use vector in trimLiterals

7 years agorose: more hash member funcs for rose types
Justin Viiret [Mon, 31 Jul 2017 06:22:08 +0000 (16:22 +1000)] 
rose: more hash member funcs for rose types

7 years agorose: return a vector from findEdgesByLiteral
Justin Viiret [Mon, 31 Jul 2017 06:02:55 +0000 (16:02 +1000)] 
rose: return a vector from findEdgesByLiteral

7 years agorose_build_long_lit: refactor to do less dupe work
Justin Viiret [Mon, 29 May 2017 04:59:31 +0000 (14:59 +1000)] 
rose_build_long_lit: refactor to do less dupe work

Also some performance improvements.

7 years agodynamic compression: add HS_CDECL to implementation
Alex Coyte [Mon, 7 Aug 2017 03:50:21 +0000 (13:50 +1000)] 
dynamic compression: add HS_CDECL to implementation

7 years agoProvide RoseResources to roseQuality.
Alex Coyte [Mon, 31 Jul 2017 00:38:30 +0000 (10:38 +1000)] 
Provide RoseResources to roseQuality.

RoseResources is an alternative to manually digging through the bytecode.

7 years agomangle fdr conf parts of scratch as well
Alex Coyte [Fri, 28 Jul 2017 04:56:54 +0000 (14:56 +1000)] 
mangle fdr conf parts of scratch as well

7 years agoroseQuality() no longer needs to be part of rose's API.
Alex Coyte [Fri, 28 Jul 2017 04:51:58 +0000 (14:51 +1000)] 
roseQuality() no longer needs to be part of rose's API.

7 years agomore comments on hwlm/fdr's start parameter
Alex Coyte [Fri, 28 Jul 2017 04:32:55 +0000 (14:32 +1000)] 
more comments on hwlm/fdr's start parameter

7 years agoRemove out of date debug output
Matthew Barr [Sun, 6 Aug 2017 23:45:28 +0000 (09:45 +1000)] 
Remove out of date debug output

7 years agoadd dynamic stream compression to fat runtime
Alex Coyte [Sun, 6 Aug 2017 23:19:23 +0000 (09:19 +1000)] 
add dynamic stream compression to fat runtime

7 years agoRemove unwanted move constructor
Matthew Barr [Fri, 16 Jun 2017 06:06:52 +0000 (16:06 +1000)] 
Remove unwanted move constructor

7 years agoclean up loops and add AVX-512
Matthew Barr [Thu, 22 Jun 2017 00:28:44 +0000 (10:28 +1000)] 
clean up loops and add AVX-512

7 years agoOpen input path once using file descriptor
Matthew Barr [Mon, 19 Jun 2017 06:27:17 +0000 (16:27 +1000)] 
Open input path once using file descriptor

7 years agoCatch by reference not value
Matthew Barr [Mon, 19 Jun 2017 02:53:34 +0000 (12:53 +1000)] 
Catch by reference not value

7 years agouse string equality operator
Matthew Barr [Fri, 16 Jun 2017 05:56:48 +0000 (15:56 +1000)] 
use string equality operator

7 years agorestore formatting flags after use
Matthew Barr [Mon, 19 Jun 2017 05:52:08 +0000 (15:52 +1000)] 
restore formatting flags after use

7 years agohsbench: use a memstream instead of a temp file
Matthew Barr [Mon, 19 Jun 2017 05:00:51 +0000 (15:00 +1000)] 
hsbench: use a memstream instead of a temp file

7 years agoadd dynamic compression to the public api
Alex Coyte [Fri, 4 Aug 2017 01:40:28 +0000 (11:40 +1000)] 
add dynamic compression to the public api

7 years agoCMake: put the stream compress files in the correct part
Alex Coyte [Fri, 4 Aug 2017 01:08:30 +0000 (11:08 +1000)] 
CMake: put the stream compress files in the correct part

7 years agodev reference documentation for stream compression
Alex Coyte [Tue, 2 May 2017 03:16:41 +0000 (13:16 +1000)] 
dev reference documentation for stream compression

7 years agomultibit compression support
Hong, Yang A [Mon, 8 May 2017 19:56:28 +0000 (15:56 -0400)] 
multibit compression support

7 years agohsbench: add stream compress functionality
Alex Coyte [Mon, 6 Mar 2017 00:58:53 +0000 (11:58 +1100)] 
hsbench: add stream compress functionality

7 years agosupport dynamic stream compression
Alex Coyte [Tue, 14 Feb 2017 03:18:13 +0000 (14:18 +1100)] 
support dynamic stream compression

7 years agomake ComponentRepeat::vacuous_everywhere() more accurate
Alex Coyte [Wed, 2 Aug 2017 03:36:24 +0000 (13:36 +1000)] 
make ComponentRepeat::vacuous_everywhere() more accurate

7 years agomergeCastle: merge common repeats from the castles
Alex Coyte [Wed, 2 Aug 2017 03:07:24 +0000 (13:07 +1000)] 
mergeCastle: merge common repeats from the castles

7 years agomergeLeftfixesVariableLag: update comments, debugging support
Alex Coyte [Wed, 2 Aug 2017 02:54:02 +0000 (12:54 +1000)] 
mergeLeftfixesVariableLag: update comments, debugging support

7 years agoCompile dump of teddy's nibble masks and reinforcement table in fdr_dump.cpp
Chang, Harry [Thu, 27 Jul 2017 09:21:05 +0000 (17:21 +0800)] 
Compile dump of teddy's nibble masks and reinforcement table in fdr_dump.cpp

7 years agosmall_vector: require boost >= 1.61
Justin Viiret [Mon, 31 Jul 2017 02:26:45 +0000 (12:26 +1000)] 
small_vector: require boost >= 1.61

We use the small_vector constructors introduced in Boost 1.61 (trac bug
11866, github commit b436c91). If the Boost version is too old, we fall
back to using std::vector.

7 years agocmake: another convenience lib for compile side
Matthew Barr [Tue, 13 Jun 2017 04:26:24 +0000 (14:26 +1000)] 
cmake: another convenience lib for compile side

7 years agoDo equivalency removal before violet's implementablity check.
Alex Coyte [Tue, 18 Jul 2017 02:49:32 +0000 (12:49 +1000)] 
Do equivalency removal before violet's implementablity check.

This is helpful as removing/restoring literals may introduce redundancy in the
graphs. Also improve the implementation by caching known good holders.

7 years agoEnsure max width of repeat before transforming graph.
Alex Coyte [Wed, 19 Jul 2017 00:02:55 +0000 (10:02 +1000)] 
Ensure max width of repeat before transforming graph.

7 years agoRefactored building reinforcement table at compile time and updated comments.
Chang, Harry [Mon, 24 Jul 2017 09:37:42 +0000 (17:37 +0800)] 
Refactored building reinforcement table at compile time and updated comments.

7 years agoreplace "_avx2" with "_fat".
Chang, Harry [Mon, 24 Jul 2017 07:09:17 +0000 (15:09 +0800)] 
replace "_avx2" with "_fat".

7 years agoAVX512 Reinforced FAT teddy.
Chang, Harry [Thu, 13 Jul 2017 06:38:06 +0000 (14:38 +0800)] 
AVX512 Reinforced FAT teddy.

7 years agodeterminise: use find first, rather than emplace
Justin Viiret [Fri, 21 Jul 2017 06:43:16 +0000 (16:43 +1000)] 
determinise: use find first, rather than emplace

For non-trivial StateSet types, copying to do the emplace if it is
already in the map is more expensive than checking with find() first.

7 years agotidy: "ue2::flat_set/map" -> "flat_set/map"
Justin Viiret [Wed, 19 Jul 2017 01:20:39 +0000 (11:20 +1000)] 
tidy: "ue2::flat_set/map" -> "flat_set/map"

7 years agoutil: switch from Boost to std::unordered set/map
Justin Viiret [Fri, 14 Jul 2017 04:59:52 +0000 (14:59 +1000)] 
util: switch from Boost to std::unordered set/map

This commit replaces the ue2::unordered_{set,map} types with their STL
versions, with some new hashing utilities in util/hash.h. The new types
ue2_unordered_set<T> and ue2_unordered_map<Key, T> default to using the
ue2_hasher.

The header util/ue2_containers.h has been removed, and the flat_set/map
containers moved to util/flat_containers.h.

7 years agoue2_graph: move descriptors out of graph struct
Justin Viiret [Fri, 14 Jul 2017 04:51:53 +0000 (14:51 +1000)] 
ue2_graph: move descriptors out of graph struct

7 years agoue-3145: make parents of included literals exclusive
Wang, Xiang W [Thu, 20 Jul 2017 20:40:54 +0000 (16:40 -0400)] 
ue-3145: make parents of included literals exclusive

7 years agoFDR: Squash buckets of included literals in FDR confirm
Wang, Xiang W [Thu, 22 Jun 2017 08:50:45 +0000 (04:50 -0400)] 
FDR: Squash buckets of included literals in FDR confirm

 - Change the compile of literal matchers to two passes.
 - Reverse the bucket assignment in FDR, bucket with longer literals has
   smaller bucket id.
 - Squash the buckets of included literals and jump to the the program of
   included literals directly from parent literal program without going
   through FDR confirm for included iterals.

7 years agofix typo "ones_u32a" => "ones_u32"
Chang, Harry [Mon, 24 Jul 2017 03:05:46 +0000 (11:05 +0800)] 
fix typo "ones_u32a" => "ones_u32"

7 years agoAVX512 reinforced teddy.
Chang, Harry [Thu, 6 Jul 2017 01:42:17 +0000 (18:42 -0700)] 
AVX512 reinforced teddy.

7 years agosmallwrite: batch dfa merge to reduce compile time
Justin Viiret [Thu, 20 Jul 2017 05:11:50 +0000 (15:11 +1000)] 
smallwrite: batch dfa merge to reduce compile time

7 years agoliteral matchers: change context passed to callback to scratch
Wang, Xiang W [Tue, 4 Jul 2017 16:24:11 +0000 (12:24 -0400)] 
literal matchers: change context passed to callback to scratch

7 years agong_find_matches: speed up edge lookups
Justin Viiret [Wed, 12 Jul 2017 01:08:45 +0000 (11:08 +1000)] 
ng_find_matches: speed up edge lookups

Improves the performance of step() on graphs with vertices with large
degree.

7 years agoflood detection: debug output fix
Wang, Xiang W [Fri, 7 Jul 2017 12:14:35 +0000 (08:14 -0400)] 
flood detection: debug output fix

7 years agoremove start argument in literal matcher callbacks
Wang, Xiang W [Thu, 6 Jul 2017 16:23:41 +0000 (12:23 -0400)] 
remove start argument in literal matcher callbacks

7 years agoEnsure that reports would be sustained after self loop is removed
Alex Coyte [Wed, 5 Jul 2017 01:00:39 +0000 (11:00 +1000)] 
Ensure that reports would be sustained after self loop is removed

Approximante matching means that is now possible to get a non-standard report
on a cyclic during edge redundancy passes which means checks are now needed.

7 years agong_prefilter.cpp: remove interior edges first
Justin Viiret [Tue, 4 Jul 2017 03:22:09 +0000 (13:22 +1000)] 
ng_prefilter.cpp: remove interior edges first

This allows us to avoid looking them up while we're in copyOutEdges(),
halves time on large cases.

7 years agong_mcclellan: reject determinise if NFA is too big
Justin Viiret [Mon, 26 Jun 2017 06:15:23 +0000 (16:15 +1000)] 
ng_mcclellan: reject determinise if NFA is too big

7 years agong_violet: fail on added_count limit quicker
Justin Viiret [Mon, 26 Jun 2017 06:17:25 +0000 (16:17 +1000)] 
ng_violet: fail on added_count limit quicker

- also fixes typo "splitForImplementability"
- adds more detail in debug output

7 years agorose_build_convert: replace bind2nd with lambda
Justin Viiret [Mon, 3 Jul 2017 01:08:25 +0000 (11:08 +1000)] 
rose_build_convert: replace bind2nd with lambda

std::bind2nd was deprecated in C++11.

7 years agordfa: move raw_dfa member functions to rdfa.cpp
Justin Viiret [Mon, 3 Jul 2017 01:29:35 +0000 (11:29 +1000)] 
rdfa: move raw_dfa member functions to rdfa.cpp

7 years agopatch for invalid reading 1 byte in Reinforced Teddy, abandon fetching the first...
Chang, Harry [Fri, 30 Jun 2017 00:13:27 +0000 (17:13 -0700)] 
patch for invalid reading 1 byte in Reinforced Teddy, abandon fetching the first reinforced byte.

7 years agocmake: add graph_range.h
Justin Viiret [Fri, 30 Jun 2017 01:15:39 +0000 (11:15 +1000)] 
cmake: add graph_range.h

7 years agong_depth: more use of small_color_map
Justin Viiret [Thu, 29 Jun 2017 23:43:31 +0000 (09:43 +1000)] 
ng_depth: more use of small_color_map

7 years agong_depth: use small_color_map in findLoopReachable
Justin Viiret [Thu, 29 Jun 2017 01:43:45 +0000 (11:43 +1000)] 
ng_depth: use small_color_map in findLoopReachable

7 years agong_util: make more use of small_color_map in DFS
Justin Viiret [Thu, 29 Jun 2017 01:38:13 +0000 (11:38 +1000)] 
ng_util: make more use of small_color_map in DFS

7 years agong_util: use small_color_map
Justin Viiret [Thu, 29 Jun 2017 01:33:03 +0000 (11:33 +1000)] 
ng_util: use small_color_map

7 years agong_prune: use small_color_map
Justin Viiret [Thu, 29 Jun 2017 01:25:13 +0000 (11:25 +1000)] 
ng_prune: use small_color_map

7 years agong_netflow: use small_color_map
Justin Viiret [Thu, 29 Jun 2017 01:20:17 +0000 (11:20 +1000)] 
ng_netflow: use small_color_map

7 years agong_literal_analysis: use small_color_map
Justin Viiret [Thu, 29 Jun 2017 01:14:46 +0000 (11:14 +1000)] 
ng_literal_analysis: use small_color_map

7 years agong_repeat: use small_color_map
Justin Viiret [Thu, 29 Jun 2017 01:08:54 +0000 (11:08 +1000)] 
ng_repeat: use small_color_map

7 years agong_region: use small_color_map
Justin Viiret [Thu, 29 Jun 2017 01:05:55 +0000 (11:05 +1000)] 
ng_region: use small_color_map

7 years agong_width: use small_color_map
Justin Viiret [Thu, 29 Jun 2017 00:57:17 +0000 (10:57 +1000)] 
ng_width: use small_color_map

7 years agong_misc_opt: use small_color_map
Justin Viiret [Thu, 29 Jun 2017 00:38:46 +0000 (10:38 +1000)] 
ng_misc_opt: use small_color_map

7 years agolimex_compile: use small_color_map
Justin Viiret [Thu, 29 Jun 2017 00:26:17 +0000 (10:26 +1000)] 
limex_compile: use small_color_map

7 years agosmall_color_map: add efficient 2-bit color map
Justin Viiret [Wed, 28 Jun 2017 06:29:31 +0000 (16:29 +1000)] 
small_color_map: add efficient 2-bit color map

7 years agonoodle: correct streaming bounds
Matthew Barr [Fri, 30 Jun 2017 01:42:32 +0000 (11:42 +1000)] 
noodle: correct streaming bounds

7 years agonoodle: param name
Matthew Barr [Thu, 29 Jun 2017 23:10:48 +0000 (09:10 +1000)] 
noodle: param name

7 years agonoodle: scan using the correct offsets
Matthew Barr [Thu, 29 Jun 2017 06:26:56 +0000 (16:26 +1000)] 
noodle: scan using the correct offsets

7 years agoshufti/truffle tests: silence ubsan warning
Justin Viiret [Thu, 25 May 2017 02:46:01 +0000 (12:46 +1000)] 
shufti/truffle tests: silence ubsan warning

The ubsan support in clang warned about us accessing idx-1 of an array here.

7 years agotamarama_dump: fix typo in comment
Justin Viiret [Thu, 29 Jun 2017 03:12:23 +0000 (13:12 +1000)] 
tamarama_dump: fix typo in comment

7 years agoslot_manager_dump: use StdioFile
Justin Viiret [Thu, 29 Jun 2017 03:20:31 +0000 (13:20 +1000)] 
slot_manager_dump: use StdioFile

7 years agosmallwrite_dump: use StdioFile
Justin Viiret [Thu, 29 Jun 2017 03:18:51 +0000 (13:18 +1000)] 
smallwrite_dump: use StdioFile

7 years agorose_in_dump: use StdioFile
Justin Viiret [Thu, 29 Jun 2017 03:17:04 +0000 (13:17 +1000)] 
rose_in_dump: use StdioFile

7 years agong_dump: use StdioFile
Justin Viiret [Thu, 29 Jun 2017 03:15:07 +0000 (13:15 +1000)] 
ng_dump: use StdioFile

7 years agotamarama_dump: use StdioFile
Justin Viiret [Thu, 29 Jun 2017 03:11:54 +0000 (13:11 +1000)] 
tamarama_dump: use StdioFile

7 years agoshengdump: use StdioFile
Justin Viiret [Thu, 29 Jun 2017 03:10:40 +0000 (13:10 +1000)] 
shengdump: use StdioFile

7 years agompv_dump: use StdioFile
Justin Viiret [Thu, 29 Jun 2017 03:09:16 +0000 (13:09 +1000)] 
mpv_dump: use StdioFile

7 years agomcsheng_dump: use StdioFile
Justin Viiret [Thu, 29 Jun 2017 03:07:39 +0000 (13:07 +1000)] 
mcsheng_dump: use StdioFile

7 years agomcclellandump: use StdioFile
Justin Viiret [Thu, 29 Jun 2017 03:06:26 +0000 (13:06 +1000)] 
mcclellandump: use StdioFile

7 years agolimex_dump: use StdioFile, tidy
Justin Viiret [Thu, 29 Jun 2017 03:03:52 +0000 (13:03 +1000)] 
limex_dump: use StdioFile, tidy

7 years agolbr_dump: use StdioFile
Justin Viiret [Thu, 29 Jun 2017 02:56:55 +0000 (12:56 +1000)] 
lbr_dump: use StdioFile

7 years agogoughdump: use StdioFile
Justin Viiret [Thu, 29 Jun 2017 02:54:26 +0000 (12:54 +1000)] 
goughdump: use StdioFile

7 years agogoughcompile_dump: use StdioFile
Justin Viiret [Thu, 29 Jun 2017 02:52:14 +0000 (12:52 +1000)] 
goughcompile_dump: use StdioFile

7 years agocastle_dump: use StdioFile
Justin Viiret [Thu, 29 Jun 2017 02:46:56 +0000 (12:46 +1000)] 
castle_dump: use StdioFile

7 years agonoodle: behave like our other literal matchers
Matthew Barr [Tue, 30 May 2017 05:54:51 +0000 (15:54 +1000)] 
noodle: behave like our other literal matchers

Noodle now supports supplementary masks.

7 years agoMove hwlm literal len define
Matthew Barr [Mon, 26 Jun 2017 00:15:49 +0000 (10:15 +1000)] 
Move hwlm literal len define