[thirdparty/systemd.git] / docs / JOURNAL_EXPORT_FORMATS.md

---
title: Journal Export Formats
category: Interfaces
layout: default
SPDX-License-Identifier: LGPL-2.1-or-later
---

# Journal Export Formats

## Journal Export Format

_Note that this document describes the binary serialization format of journals only, as used for transfer across the network.
For interfacing with web technologies there's the Journal JSON Format, described below.
The binary format on disk is documented as the [Journal File Format](JOURNAL_FILE_FORMAT)._

_Before reading on, please make sure you are aware of the [basic properties of journal entries](https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html), in particular realize that they may include binary non-text data (though usually don't), and the same field might have multiple values assigned within the same entry (though usually hasn't)._

When exporting journal data for other uses or transferring it via the network/local IPC the _journal export format_ is used. It's a simple serialization of journal entries, that is easy to read without any special tools, but still binary safe where necessary. The format is like this:

* Two journal entries that follow each other are separated by a double newline.
* Journal fields consisting only of valid non-control UTF-8 codepoints are serialized as they are (i.e. the field name, followed by '=', followed by field data), followed by a newline as separator to the next field. Note that fields containing newlines cannot be formatted like this. Non-control UTF-8 codepoints are the codepoints with value at or above 32 (' '), or equal to 9 (TAB).
* Other journal fields are serialized in a special binary safe way: field name, followed by newline, followed by a binary 64-bit little endian size value, followed by the binary field data, followed by a newline as separator to the next field.
* Entry metadata that is not actually a field is serialized like it was a field, but beginning with two underscores. More specifically, `__CURSOR=`, `__REALTIME_TIMESTAMP=`, `__MONOTONIC_TIMESTAMP=`, `__SEQNUM=`, `__SEQNUM_ID` are introduced this way. Note that these meta-fields are only generated when actual journal files are serialized. They are omitted for entries that do not originate from a journal file (for example because they are transferred for the first time to be stored in one). Or in other words: if you are generating this format you shouldn't care about these special double-underscore fields. But you might find them usable when you deserialize the format generated by us. Additional fields prefixed with two underscores might be added later on, your parser should skip over the fields it does not know.
* The order in which fields appear in an entry is undefined and might be different for each entry that is serialized.
And that's already it.

This format can be generated via `journalctl -o export`.

Here's an example for two serialized entries which consist only of text data:

```
__CURSOR=s=739ad463348b4ceca5a9e69c95a3c93f;i=4ece7;b=6c7c6013a26343b29e964691ff25d04c;m=4fc72436e;t=4c508a72423d9;x=d3e5610681098c10;p=system.journal
__REALTIME_TIMESTAMP=1342540861416409
__MONOTONIC_TIMESTAMP=21415215982
_BOOT_ID=6c7c6013a26343b29e964691ff25d04c
_TRANSPORT=syslog
PRIORITY=4
SYSLOG_FACILITY=3
SYSLOG_IDENTIFIER=gdm-password]
SYSLOG_PID=587
MESSAGE=AccountsService-DEBUG(+): ActUserManager: ignoring unspecified session '8' since it's not graphical: Success
_PID=587
_UID=0
_GID=500
_COMM=gdm-session-wor
_EXE=/usr/libexec/gdm-session-worker
_CMDLINE=gdm-session-worker [pam/gdm-password]
_AUDIT_SESSION=2
_AUDIT_LOGINUID=500
_SYSTEMD_CGROUP=/user/lennart/2
_SYSTEMD_SESSION=2
_SELINUX_CONTEXT=system_u:system_r:xdm_t:s0-s0:c0.c1023
_SOURCE_REALTIME_TIMESTAMP=1342540861413961
_MACHINE_ID=a91663387a90b89f185d4e860000001a
_HOSTNAME=epsilon

__CURSOR=s=739ad463348b4ceca5a9e69c95a3c93f;i=4ece8;b=6c7c6013a26343b29e964691ff25d04c;m=4fc72572f;t=4c508a7243799;x=68597058a89b7246;p=system.journal
__REALTIME_TIMESTAMP=1342540861421465
__MONOTONIC_TIMESTAMP=21415221039
_BOOT_ID=6c7c6013a26343b29e964691ff25d04c
_TRANSPORT=syslog
PRIORITY=6
SYSLOG_FACILITY=9
SYSLOG_IDENTIFIER=/USR/SBIN/CROND
SYSLOG_PID=8278
MESSAGE=(root) CMD (run-parts /etc/cron.hourly)
_PID=8278
_UID=0
_GID=0
_COMM=run-parts
_EXE=/usr/bin/bash
_CMDLINE=/bin/bash /bin/run-parts /etc/cron.hourly
_AUDIT_SESSION=8
_AUDIT_LOGINUID=0
_SYSTEMD_CGROUP=/user/root/8
_SYSTEMD_SESSION=8
_SELINUX_CONTEXT=system_u:system_r:crond_t:s0-s0:c0.c1023
_SOURCE_REALTIME_TIMESTAMP=1342540861416351
_MACHINE_ID=a91663387a90b89f185d4e860000001a
_HOSTNAME=epsilon

```

A message with a binary field produced by
```bash
python3 -c 'from systemd import journal; journal.send("foo\nbar")'
journalctl -n1 -o export
```

```
__CURSOR=s=bcce4fb8ffcb40e9a6e05eee8b7831bf;i=5ef603;b=ec25d6795f0645619ddac9afdef453ee;m=545242e7049;t=50f1202
__REALTIME_TIMESTAMP=1423944916375353
__MONOTONIC_TIMESTAMP=5794517905481
_BOOT_ID=ec25d6795f0645619ddac9afdef453ee
_TRANSPORT=journal
_UID=1001
_GID=1001
_CAP_EFFECTIVE=0
_SYSTEMD_OWNER_UID=1001
_SYSTEMD_SLICE=user-1001.slice
_MACHINE_ID=5833158886a8445e801d437313d25eff
_HOSTNAME=bupkis
_AUDIT_LOGINUID=1001
_SELINUX_CONTEXT=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
CODE_LINE=1
CODE_FUNC=<module>
SYSLOG_IDENTIFIER=python3
_COMM=python3
_EXE=/usr/bin/python3.4
_AUDIT_SESSION=35898
_SYSTEMD_CGROUP=/user.slice/user-1001.slice/session-35898.scope
_SYSTEMD_SESSION=35898
_SYSTEMD_UNIT=session-35898.scope
MESSAGE
^G^@^@^@^@^@^@^@foo
bar
CODE_FILE=<string>
_PID=16853
_CMDLINE=python3 -c from systemd import journal; journal.send("foo\nbar")
_SOURCE_REALTIME_TIMESTAMP=1423944916372858
```

## Journal JSON Format

_Note that this section describes the JSON serialization format of the journal only, as used for interfacing with web technologies.
For binary transfer of journal data across the network there's the Journal Export Format described above.
The binary format on disk is documented as [Journal File Format](JOURNAL_FILE_FORMAT)._

_Before reading on, please make sure you are aware of the [basic properties of journal entries](https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html), in particular realize that they may include binary non-text data (though usually don't), and the same field might have multiple values assigned within the same entry (though usually hasn't)._

In most cases the Journal JSON serialization is the obvious mapping of the entry field names (as JSON strings) to the entry field values (also as JSON strings) encapsulated in one JSON object. However, there are a few special cases to handle:

* A field that contains non-printable or non-UTF8 is serialized as a number array instead. This is necessary to handle binary data in a safe way without losing data, since JSON cannot embed binary data natively. Each byte of the binary field will be mapped to its numeric value in the range 0…255.
* The JSON serializer can optionally skip huge (as in larger than a specific threshold) data fields from the JSON object. If that is enabled and a data field is too large, the field name is still included in the JSON object but assigned _null_.
* Within the same entry, Journal fields may have multiple values assigned. This is not allowed in JSON. The serializer will hence create a single JSON field only for these cases, and assign it an array of values (which the can be strings, _null_ or number arrays, see above).
* If the JSON data originates from a journal file it may include the special addressing fields `__CURSOR`, `__REALTIME_TIMESTAMP`, `__MONOTONIC_TIMESTAMP`, `__SEQNUM`, `__SEQNUM_ID`, which contain the cursor string of this entry as string, the realtime/monotonic timestamps of this entry as formatted numeric string of usec since the respective epoch, and the sequence number and associated sequence number ID, both formatted as strings.

Here's an example, illustrating all cases mentioned above. Consider this entry:

```
MESSAGE=Hello World
_UDEV_DEVNODE=/dev/waldo
_UDEV_DEVLINK=/dev/alias1
_UDEV_DEVLINK=/dev/alias2
BINARY=this is a binary value \a
LARGE=this is a super large value (let's pretend at least, for the sake of this example)
```

This translates into the following JSON Object:
```json
{
  "MESSAGE" : "Hello World",
  "_UDEV_DEVNODE" : "/dev/waldo",
  "_UDEV_DEVLINK" : [ "/dev/alias1", "/dev/alias2" ],
  "BINARY" : [ 116, 104, 105, 115, 32, 105, 115, 32, 97, 32, 98, 105, 110, 97, 114, 121, 32, 118, 97, 108, 117, 101, 32, 7 ],
  "LARGE" : null
}
```
Commit	Line	Data
5e3ab38e	1	---
63812e15	2	title: Journal Export Formats
5e3ab38e ZJS	3	category: Interfaces
	4	layout: default
	5	SPDX-License-Identifier: LGPL-2.1-or-later
	6	---
	7
63812e15 BF	8	# Journal Export Formats
	9
	10	## Journal Export Format
5e3ab38e ZJS	11
5e3ab38e ZJS	12	_Note that this document describes the binary serialization format of journals only, as used for transfer across the network.
d9044a43	13	For interfacing with web technologies there's the Journal JSON Format, described below.
1b4dc2ea	14	The binary format on disk is documented as the [Journal File Format](JOURNAL_FILE_FORMAT)._
5e3ab38e ZJS	15
	16	_Before reading on, please make sure you are aware of the [basic properties of journal entries](https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html), in particular realize that they may include binary non-text data (though usually don't), and the same field might have multiple values assigned within the same entry (though usually hasn't)._
	17
	18	When exporting journal data for other uses or transferring it via the network/local IPC the _journal export format_ is used. It's a simple serialization of journal entries, that is easy to read without any special tools, but still binary safe where necessary. The format is like this:
	19
	20	* Two journal entries that follow each other are separated by a double newline.
	21	* Journal fields consisting only of valid non-control UTF-8 codepoints are serialized as they are (i.e. the field name, followed by '=', followed by field data), followed by a newline as separator to the next field. Note that fields containing newlines cannot be formatted like this. Non-control UTF-8 codepoints are the codepoints with value at or above 32 (' '), or equal to 9 (TAB).
da890466	22	* Other journal fields are serialized in a special binary safe way: field name, followed by newline, followed by a binary 64-bit little endian size value, followed by the binary field data, followed by a newline as separator to the next field.
11181f8a	23	* Entry metadata that is not actually a field is serialized like it was a field, but beginning with two underscores. More specifically, `__CURSOR=`, `__REALTIME_TIMESTAMP=`, `__MONOTONIC_TIMESTAMP=`, `__SEQNUM=`, `__SEQNUM_ID` are introduced this way. Note that these meta-fields are only generated when actual journal files are serialized. They are omitted for entries that do not originate from a journal file (for example because they are transferred for the first time to be stored in one). Or in other words: if you are generating this format you shouldn't care about these special double-underscore fields. But you might find them usable when you deserialize the format generated by us. Additional fields prefixed with two underscores might be added later on, your parser should skip over the fields it does not know.
5e3ab38e ZJS	24	* The order in which fields appear in an entry is undefined and might be different for each entry that is serialized.
	25	And that's already it.
	26
	27	This format can be generated via `journalctl -o export`.
	28
	29	Here's an example for two serialized entries which consist only of text data:
	30
	31	```
	32	__CURSOR=s=739ad463348b4ceca5a9e69c95a3c93f;i=4ece7;b=6c7c6013a26343b29e964691ff25d04c;m=4fc72436e;t=4c508a72423d9;x=d3e5610681098c10;p=system.journal
	33	__REALTIME_TIMESTAMP=1342540861416409
	34	__MONOTONIC_TIMESTAMP=21415215982
	35	_BOOT_ID=6c7c6013a26343b29e964691ff25d04c
	36	_TRANSPORT=syslog
	37	PRIORITY=4
	38	SYSLOG_FACILITY=3
	39	SYSLOG_IDENTIFIER=gdm-password]
	40	SYSLOG_PID=587
	41	MESSAGE=AccountsService-DEBUG(+): ActUserManager: ignoring unspecified session '8' since it's not graphical: Success
	42	_PID=587
	43	_UID=0
	44	_GID=500
	45	_COMM=gdm-session-wor
	46	_EXE=/usr/libexec/gdm-session-worker
	47	_CMDLINE=gdm-session-worker [pam/gdm-password]
	48	_AUDIT_SESSION=2
	49	_AUDIT_LOGINUID=500
	50	_SYSTEMD_CGROUP=/user/lennart/2
	51	_SYSTEMD_SESSION=2
	52	_SELINUX_CONTEXT=system_u:system_r:xdm_t:s0-s0:c0.c1023
	53	_SOURCE_REALTIME_TIMESTAMP=1342540861413961
	54	_MACHINE_ID=a91663387a90b89f185d4e860000001a
	55	_HOSTNAME=epsilon
	56
	57	__CURSOR=s=739ad463348b4ceca5a9e69c95a3c93f;i=4ece8;b=6c7c6013a26343b29e964691ff25d04c;m=4fc72572f;t=4c508a7243799;x=68597058a89b7246;p=system.journal
	58	__REALTIME_TIMESTAMP=1342540861421465
	59	__MONOTONIC_TIMESTAMP=21415221039
	60	_BOOT_ID=6c7c6013a26343b29e964691ff25d04c
	61	_TRANSPORT=syslog
	62	PRIORITY=6
	63	SYSLOG_FACILITY=9
	64	SYSLOG_IDENTIFIER=/USR/SBIN/CROND
	65	SYSLOG_PID=8278
	66	MESSAGE=(root) CMD (run-parts /etc/cron.hourly)
	67	_PID=8278
	68	_UID=0
	69	_GID=0
	70	_COMM=run-parts
	71	_EXE=/usr/bin/bash
	72	_CMDLINE=/bin/bash /bin/run-parts /etc/cron.hourly
	73	_AUDIT_SESSION=8
	74	_AUDIT_LOGINUID=0
	75	_SYSTEMD_CGROUP=/user/root/8
	76	_SYSTEMD_SESSION=8
	77	_SELINUX_CONTEXT=system_u:system_r:crond_t:s0-s0:c0.c1023
	78	_SOURCE_REALTIME_TIMESTAMP=1342540861416351
	79	_MACHINE_ID=a91663387a90b89f185d4e860000001a
	80	_HOSTNAME=epsilon
	81
	82	```
	83
	84	A message with a binary field produced by
	85	```bash
	86	python3 -c 'from systemd import journal; journal.send("foo\nbar")'
	87	journalctl -n1 -o export
88	```
89
90	```
91	__CURSOR=s=bcce4fb8ffcb40e9a6e05eee8b7831bf;i=5ef603;b=ec25d6795f0645619ddac9afdef453ee;m=545242e7049;t=50f1202
92	__REALTIME_TIMESTAMP=1423944916375353
93	__MONOTONIC_TIMESTAMP=5794517905481
94	_BOOT_ID=ec25d6795f0645619ddac9afdef453ee
95	_TRANSPORT=journal
96	_UID=1001
97	_GID=1001
98	_CAP_EFFECTIVE=0
99	_SYSTEMD_OWNER_UID=1001
100	_SYSTEMD_SLICE=user-1001.slice
101	_MACHINE_ID=5833158886a8445e801d437313d25eff
102	_HOSTNAME=bupkis
103	_AUDIT_LOGINUID=1001
104	_SELINUX_CONTEXT=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
105	CODE_LINE=1
106	CODE_FUNC=<module>
107	SYSLOG_IDENTIFIER=python3
108	_COMM=python3
109	_EXE=/usr/bin/python3.4
110	_AUDIT_SESSION=35898
111	_SYSTEMD_CGROUP=/user.slice/user-1001.slice/session-35898.scope
112	_SYSTEMD_SESSION=35898
113	_SYSTEMD_UNIT=session-35898.scope
114	MESSAGE
115	^G^@^@^@^@^@^@^@foo
116	bar
117	CODE_FILE=<string>
118	_PID=16853
119	_CMDLINE=python3 -c from systemd import journal; journal.send("foo\nbar")
120	_SOURCE_REALTIME_TIMESTAMP=1423944916372858
121	```
d9044a43	122
63812e15	123	## Journal JSON Format
d9044a43 ZJS	124
	125	_Note that this section describes the JSON serialization format of the journal only, as used for interfacing with web technologies.
	126	For binary transfer of journal data across the network there's the Journal Export Format described above.
1b4dc2ea	127	The binary format on disk is documented as [Journal File Format](JOURNAL_FILE_FORMAT)._
d9044a43 ZJS	128
	129	_Before reading on, please make sure you are aware of the [basic properties of journal entries](https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html), in particular realize that they may include binary non-text data (though usually don't), and the same field might have multiple values assigned within the same entry (though usually hasn't)._
	130
	131	In most cases the Journal JSON serialization is the obvious mapping of the entry field names (as JSON strings) to the entry field values (also as JSON strings) encapsulated in one JSON object. However, there are a few special cases to handle:
	132
	133	* A field that contains non-printable or non-UTF8 is serialized as a number array instead. This is necessary to handle binary data in a safe way without losing data, since JSON cannot embed binary data natively. Each byte of the binary field will be mapped to its numeric value in the range 0…255.
	134	* The JSON serializer can optionally skip huge (as in larger than a specific threshold) data fields from the JSON object. If that is enabled and a data field is too large, the field name is still included in the JSON object but assigned _null_.
	135	* Within the same entry, Journal fields may have multiple values assigned. This is not allowed in JSON. The serializer will hence create a single JSON field only for these cases, and assign it an array of values (which the can be strings, _null_ or number arrays, see above).
11181f8a	136	* If the JSON data originates from a journal file it may include the special addressing fields `__CURSOR`, `__REALTIME_TIMESTAMP`, `__MONOTONIC_TIMESTAMP`, `__SEQNUM`, `__SEQNUM_ID`, which contain the cursor string of this entry as string, the realtime/monotonic timestamps of this entry as formatted numeric string of usec since the respective epoch, and the sequence number and associated sequence number ID, both formatted as strings.
d9044a43 ZJS	137
	138	Here's an example, illustrating all cases mentioned above. Consider this entry:
	139
	140	```
	141	MESSAGE=Hello World
	142	_UDEV_DEVNODE=/dev/waldo
	143	_UDEV_DEVLINK=/dev/alias1
	144	_UDEV_DEVLINK=/dev/alias2
	145	BINARY=this is a binary value \a
	146	LARGE=this is a super large value (let's pretend at least, for the sake of this example)
	147	```
	148
	149	This translates into the following JSON Object:
	150	```json
	151	{
	152	"MESSAGE" : "Hello World",
	153	"_UDEV_DEVNODE" : "/dev/waldo",
	154	"_UDEV_DEVLINK" : [ "/dev/alias1", "/dev/alias2" ],
	155	"BINARY" : [ 116, 104, 105, 115, 32, 105, 115, 32, 97, 32, 98, 105, 110, 97, 114, 121, 32, 118, 97, 108, 117, 101, 32, 7 ],
	156	"LARGE" : null
	157	}
	158	```