This section describes the set of utilities included with the Hyperscan library.
+********************
+Quick Check: hscheck
+********************
+
+The ``hscheck`` tool allows the user to quickly check whether Hyperscan supports
+a group of patterns. If a pattern is rejected by Hyperscan's compiler, the
+compile error is provided on standard output.
+
+For example, given the following three patterns (the last of which contains a
+syntax error) in a file called ``/tmp/test``::
+
+ 1:/foo.*bar/
+ 2:/abc|def|ghi/
+ 3:/((foo|bar)/
+
+... the ``hscheck`` tool will produce the following output::
+
+ $ bin/hscheck -e /tmp/test
+
+ OK: 1:/foo.*bar/
+ OK: 2:/abc|def|ghi/
+ FAIL (compile): 3:/((foo|bar)/: Missing close parenthesis for group started at index 0.
+ SUMMARY: 1 of 3 failed.
+
********************
Benchmarker: hsbench
********************
using a utility like ``taskset`` to lock the hsbench process to one core and
minimize jitter due to the operating system's scheduler.
+*******************************
+Correctness Testing: hscollider
+*******************************
+
+The ``hscollider`` tool, or Pattern Collider, provides a way to verify
+Hyperscan's matching behaviour. It does this by compiling and scanning patterns
+(either singly or in groups) against known corpora and comparing the results
+against another engine (the "ground truth"). Two sources of ground truth for
+comparison are available:
+
+ * The PCRE library (http://pcre.org/).
+ * An NFA simulation run on Hyperscan's compile-time graph representation. This
+ is used if PCRE cannot support the pattern or if PCRE execution fails due to
+ a resource limit.
+
+Much of Hyperscan's testing infrastructure is built on ``hscollider``, and the
+tool is designed to take advantage of multiple cores and provide considerable
+flexibility in controlling the test. These options are described in the help
+(``hscollider -h``) and include:
+
+ * Testing in streaming, block or vectored mode.
+ * Testing corpora at different alignments in memory.
+ * Testing patterns in groups of varying size.
+ * Manipulating stream state or scratch space between tests.
+ * Cross-compilation and serialization/deserialization of databases.
+ * Synthetic generation of corpora given a pattern set.
+
+Using hscollider to debug a pattern
+===================================
+
+One common use-case for ``hscollider`` is to determine whether Hyperscan will
+match a pattern in the expected location, and whether this accords with PCRE's
+behaviour for the same case.
+
+Here is an example. We put our pattern in a file in Hyperscan's pattern
+format::
+
+ $ cat /tmp/pat
+ 1:/hatstand.*badgerbrush/
+
+We put the corpus to be scanned in another file, with the same numeric
+identifier at the start to indicate that it should match pattern 1::
+
+ $ cat /tmp/corpus
+ 1:__hatstand__hatstand__badgerbrush_badgerbrush
+
+Then we can run ``hscollider`` with its verbosity turned up (``-vv``) so that
+individual matches are displayed in the output::
+
+ $ bin/ue2collider -e /tmp/pat -c /tmp/corpus -Z 0 -T 1 -vv
+ ue2collider: The Pattern Collider Mark II
+
+ Number of threads: 1 (1 scanner, 1 generator)
+ Expression path: /tmp/pat
+ Signature files: none
+ Mode of operation: block mode
+ UE2 scan alignment: 0
+ Corpora read from file: /tmp/corpus
+
+ Running single-pattern/single-compile test for 1 expressions.
+
+ PCRE Match @ (2,45)
+ PCRE Match @ (2,33)
+ PCRE Match @ (12,45)
+ PCRE Match @ (12,33)
+ UE2 Match @ (0,33) for 1
+ UE2 Match @ (0,45) for 1
+ Scan call returned 0
+ PASSED: id 1, alignment 0, corpus 0 (matched pcre:2, ue2:2)
+ Thread 0 processed 1 units.
+
+ Summary:
+ Mode: Single/Block
+ =========
+ Expressions processed: 1
+ Corpora processed: 1
+ Expressions with failures: 0
+ Corpora generation failures: 0
+ Compilation failures: pcre:0, ng:0, ue2:0
+ Matching failures: pcre:0, ng:0, ue2:0
+ Match differences: 0
+ No ground truth: 0
+ Total match differences: 0
+
+ Total elapsed time: 0.00522815 secs.
+
+We can see from this output that both PCRE and Hyperscan find matches ending at
+offset 33 and 45, and so ``hscollider`` considers this test case to have
+passed.
+
+(In the example command line above, ``-Z 0`` instructs us to only test at
+corpus alignment 0, and ``-T 1`` instructs us to only use one thread.)
+
+.. note:: In default operation, PCRE produces only one match for a scan, unlike
+ Hyperscan's automata semantics. The ``hscollider`` tool uses libpcre's
+ "callout" functionality to match Hyperscan's semantics.
+
+Running a larger scan test
+==========================
+
+A set of patterns for testing purposes are distributed with Hyperscan, and these
+can be tested via ``hscollider`` on an in-tree build. Two CMake targets are
+provided to do this easily:
+
+================================= =====================================
+Make Target Description
+================================= =====================================
+``make collide_quick_test`` Tests all patterns in streaming mode.
+``make collide_quick_test_block`` Tests all patterns in block mode.
+================================= =====================================
+
+*****************
+Debugging: hsdump
+*****************
+
+When built in debug mode (using the CMake directive ``CMAKE_BUILD_TYPE`` set to
+``Debug``), Hyperscan includes support for dumping information about its
+internals during pattern compilation with the ``hsdump`` tool.
+
+This information is mostly of use to Hyperscan developers familiar with the
+library's internal structure, but can be used to diagnose issues with patterns
+and provide more information in bug reports.
+
.. _tools_pattern_format:
**************
projects / thirdparty / vectorscan.git / commitdiff
