From: Russ Combs (rucombs) <rucombs@cisco.com>
Date: Thu, 13 Apr 2017 18:02:22 +0000 (-0400)
Subject: Merge pull request #857 in SNORT/snort3 from doc_fbs to master
X-Git-Tag: 3.0.0-233~31
X-Git-Url: http://git.ipfire.org/gitweb.cgi?a=commitdiff_plain;h=97e2bcdfd03f1a2740d5571ee8fb32d626384ade;p=thirdparty%2Fsnort3.git

Merge pull request #857 in SNORT/snort3 from doc_fbs to master

Squashed commit of the following:

commit 65ddd6d2f2ace0e3a84ec6a4034095cb5bb1e4e0
Author: Carter Waxman <cwaxman@cisco.com>
Date:   Wed Apr 12 12:37:05 2017 -0400

    documented usage of flatbuffers in perfmon and file format produced
---

diff --git a/doc/perf_monitor.txt b/doc/perf_monitor.txt
index 7cebe7047..be65f9c1a 100644
--- a/doc/perf_monitor.txt
+++ b/doc/perf_monitor.txt
@@ -91,3 +91,15 @@ This tracker monitors the CPU and wall time spent by a given processing thread.
 To enable:
 
     perf_monitor = { cpu = true }
+
+==== Formatters
+
+Performance monitor allows statistics to be output in a few formats. Along with
+human readable text (as seen at shutdown) and csv formats, a Flatbuffers binary
+format is also available if Flatbuffers is present at build. A utility for
+accessing the statistics generated in this format has been included for
+convenience (see fbstreamer in tools). This tool generates a YAML array of
+records found, allowing the data to be read by humans or passed into other
+analysis tools. For information on working direclty with the Flatbuffers file
+format used by Performance monitor, see the developer notes for Performance
+monitor or the code provided for fbstreamer.
diff --git a/src/network_inspectors/perf_monitor/dev_notes.txt b/src/network_inspectors/perf_monitor/dev_notes.txt
index 0dab9e0bf..51225c32a 100644
--- a/src/network_inspectors/perf_monitor/dev_notes.txt
+++ b/src/network_inspectors/perf_monitor/dev_notes.txt
@@ -38,3 +38,41 @@ Currently output formats are:
 2. CSV
 
 3. Flatbuffers (if the library is available at build)
+
+==== Flatbuffers Parsing
+
+While a tool has been included to parse the file format used, it may be
+desired to build analysis tools that deal with the files directly. This can
+be accomplished in a few ways; either the records can be split and passed to
+flatc along with the included schema, or the schema can be parsed using the
+Flatbuffers reflection API and used to read the data within the records.
+The format for the files is as follows:
+
+===== File Layout
+
+[options="header"]
+|============================================================================
+|Section     |Description
+|File Header |File ID and schema for parsing. This appears once at the start.
+|Records     |This is a stream of records. There may be an unlimited number.
+|============================================================================
+
+===== File Header
+
+[options="header"]
+|==========================================================================
+|Field Name  |Size                |Description
+|Magic       |4 bytes             |Format identifier. Only "FLTI" exists.
+|Schema Size |4 bytes             |Size of the included schema.
+|Schema      |(schema size) bytes |Schema for parsing records in this file.
+|==========================================================================
+
+===== Record
+
+[options="header"]
+|===========================================================================
+|Field Name  |Size                |Description
+|Timestamp   |8 bytes             |The time this record was written
+|Record Size |4 bytes             |Size of the record to follow
+|Record      |(record size) bytes |Binary record. Parse against file schema.
+|===========================================================================