Apache Module mod_firehose

Available Languages: en

+ + + +

Description:	Multiplexes all I/O to a given file or pipe.
Status:	Extension
Module Identifier:	firehose_module
Source File:	mod_firehose.c

Summary

+ +

mod_firehose provides a mechanism to record data + being passed between the httpd server and the client at the raw + connection level to either a file or a pipe in such a way that the + data can be analysed or played back to the server at a future date. + It can be thought of as "tcpdump for httpd".

+ +

Connections are recorded after the SSL has been stripped, and can + be used for forensic debugging.

+ +

The firehose tool can be used to demultiplex + the recorded stream back into individual files for analysis, or + playback using a tool like netcat.

+ +

WARNING

This module IGNORES all request level + mechanisms to keep data private. It is the responsibility of the + administrator to ensure that private data is not inadvertently + exposed using this module. +

+ +

Topics

Enabling a Firehose
Stream Format

Enabling a Firehose

+ + +

To enable the module, it should be compiled and loaded + in to your running Apache configuration, and the directives below + used to record the data you are interested in.

+ +

It is possible to record both incoming and outgoing data to + the same filename if desired, as the direction of flow is recorded + within each fragment.

+ +

It is possible to write to both normal files and fifos (pipes). + In the case of fifos, mod_firehose ensures that the packet size is + no larger than PIPE_BUF to ensure writes are atomic.

+ +

If a pipe is being used, something must be reading from the pipe + before httpd is started for the pipe to be successfully opened for + write. If the request to open the pipe fails, mod_firehose will + silently stand down and not record anything, and the server will + keep running as normal.

+ +

All file writes are non blocking, and buffer overflows will cause + debugging data to be lost. The module makes the call that the + reliable running of the server takes precedence over the recording + of firehose data.

+ +

Stream Format

+ + +

The server typically serves multiple connections simultaneously, + and as a result requests and responses need to be multiplexed before + being written to the firehose.

+ +

The fragment format is designed as clear text, so that a firehose + can be opened with and inspected by a normal text editor. + Alternatively, the firehose tool can be used to + demultiplex the firehose back into individual requests or + connections.

+ +

The size of the multiplexed fragments is governed by PIPE_BUF, + the maximum size of write the system is prepared to perform + atomically. By keeping the multiplexed fragments below PIPE_BUF in + size, the module guarantees that data from different fragments does + not interleave. The size of PIPE_BUF varies on different operating + systems.

+ +

The BNF for the fragment format is as follows:

+ +

+ stream = 0*(fragment)
+
+ fragment = header CRLF body CRLF
+
+ header = length SPC timestamp SPC ( request | response ) SPC uuid SPC count
+
+ length = <up to 16 byte hex fragment length>
+ timestamp = <up to 16 byte hex timestamp microseconds since 1970>
+ request = "<"
+ response = ">"
+ uuid = <formatted uuid of the connection>
+ count = <hex fragment number in the connection>
+
+ body = <the binary content of the fragment>
+
+ SPC = <a single space>
+ CRLF = <a carriage return, followed by a line feed>
+

+ +

All fragments for a connection or a request will share the same + UUID, depending on whether connections or requests are being recorded. + If connections are being recorded, multiple requests may appear within + a connection. A fragment with a zero length indicates the end of the + connection.

+ +

Fragments may go missing or be dropped if the process reading the + fragments is too slow. If this happens, gaps will exist in the + connection counter numbering. A warning will be logged in the error + log to indicate the UUID and counter of the dropped fragment, so it + can be confirmed the fragment was dropped.

+ +

It is possible that the terminating empty fragment may not appear, + caused by the httpd process crashing, or being terminated ungracefully. + The terminating fragment may be dropped if the process reading the + fragments is not fast enough.

+ +

FirehoseConnectionInput Directive

+ + + + + + + + +

Description:	Capture traffic coming into the server on each connection
Syntax:	`FirehoseConnectionInput filename`
Default:	`none`
Context:	server config
Status:	Extension
Module:	mod_firehose
Compatibility:	FirehoseConnectionInput is only available in Apache 2.5.0 and +later.

Capture traffic coming into the server on each connection. Multiple + requests will be captured within the same connection if keepalive is + present.

+ +

Example

+ FirehoseConnectionInput connection-input.firehose +

+ +

FirehoseConnectionOutput Directive

+ + + + + + + + +

Description:	Capture traffic going out of the server on each connection
Syntax:	`FirehoseConnectionOutput filename`
Default:	`none`
Context:	server config
Status:	Extension
Module:	mod_firehose
Compatibility:	FirehoseConnectionOutput is only available in Apache 2.5.0 and +later.

Capture traffic going out of the server on each connection. + Multiple requests will be captured within the same connection if + keepalive is present.

+ +

Example

+ FirehoseConnectionOutput connection-output.firehose +

+ +

FirehoseProxyConnectionInput Directive

+ + + + + + + + +

Description:	Capture traffic coming into the back of mod_proxy
Syntax:	`FirehoseProxyConnectionInput filename`
Default:	`none`
Context:	server config
Status:	Extension
Module:	mod_firehose
Compatibility:	FirehoseProxyConnectionInput is only available in Apache 2.5.0 and +later.

Capture traffic being received by mod_proxy.

+ +

Example

+ FirehoseProxyConnectionInput proxy-input.firehose +

+ +

FirehoseProxyConnectionOutput Directive

+ + + + + + + + +

Description:	Capture traffic sent out from the back of mod_proxy
Syntax:	`FirehoseProxyConnectionOutput filename`
Default:	`none`
Context:	server config
Status:	Extension
Module:	mod_firehose
Compatibility:	FirehoseProxyConnectionOutput is only available in Apache 2.5.0 and +later.

Capture traffic being sent out by mod_proxy.

+ +

Example

+ FirehoseProxyConnectionOutput proxy-output.firehose +

+ +

FirehoseRequestInput Directive

+ + + + + + + + +

Description:	Capture traffic coming into the server on each request
Syntax:	`FirehoseRequestInput filename`
Default:	`none`
Context:	server config
Status:	Extension
Module:	mod_firehose
Compatibility:	FirehoseRequestInput is only available in Apache 2.5.0 and +later.

Capture traffic coming into the server on each request. Requests + will be captured separately, regardless of the presence of keepalive.

+ +

Example

+ FirehoseRequestInput request-input.firehose +

+ +

FirehoseRequestOutput Directive

+ + + + + + + + +

Description:	Capture traffic going out of the server on each request
Syntax:	`FirehoseRequestOutput filename`
Default:	`none`
Context:	server config
Status:	Extension
Module:	mod_firehose
Compatibility:	FirehoseRequestOutput is only available in Apache 2.5.0 and +later.

Capture traffic going out of the server on each request. Requests + will be captured separately, regardless of the presence of keepalive.

+ +

Example

+ FirehoseRequestOutput request-output.firehose +

+ +

firehose - Demultiplex a firehose stream

Available Languages: en

+ +

firehose demultiplexes the given stream of multiplexed + connections, and writes each connection to an individual file.

+ +

When writing to files, each connection is placed into a dedicated file + named after the UUID of the connection within the stream. Separate files + will be created if requests and responses are found in the stream.

+ +

If an optional prefix is specified as a parameter, connections that + start with the given prefix will be included. The prefix needs to fit + completely within the first fragment for a successful match to occur.

+ +

Synopsis
Options

Synopsis

firehose + [ -f input ] + [ -o output-directory ] + [ -u uuid ] + [ -h ] + [ --version ] + [prefix1 [...]]

+ +

Options

--file, -f filename: File to read the firehose from. Defaults to stdin.
--output-directory, -o output-directory: Directory to write demultiplexed connections to.
--uuid, -u uuid: The UUID of the connection to demultiplex. Can be specified more + than once. If not specified, all UUIDs will be demultiplexed.
--help, -h: This help text.
--version: Display the version of the program.

+ +

Apache Module mod_firehose

Summary

WARNING

Directives

Topics

Enabling a Firehose

Stream Format

FirehoseConnectionInput Directive

Example

FirehoseConnectionOutput Directive

Example

FirehoseProxyConnectionInput Directive

Example

FirehoseProxyConnectionOutput Directive

Example

FirehoseRequestInput Directive

Example

FirehoseRequestOutput Directive

Example

Overview of new features in Apache HTTP Server 2.4

firehose - Demultiplex a firehose stream

See also

Synopsis

Options