]>
Commit | Line | Data |
---|---|---|
56e6d5c0 MCC |
1 | .. SPDX-License-Identifier: GPL-2.0 |
2 | ||
3 | ================================== | |
e88d78f6 TZ |
4 | relay interface (formerly relayfs) |
5 | ================================== | |
6 | ||
7 | The relay interface provides a means for kernel applications to | |
8 | efficiently log and transfer large quantities of data from the kernel | |
9 | to userspace via user-defined 'relay channels'. | |
10 | ||
11 | A 'relay channel' is a kernel->user data relay mechanism implemented | |
12 | as a set of per-cpu kernel buffers ('channel buffers'), each | |
13 | represented as a regular file ('relay file') in user space. Kernel | |
14 | clients write into the channel buffers using efficient write | |
15 | functions; these automatically log into the current cpu's channel | |
16 | buffer. User space applications mmap() or read() from the relay files | |
17 | and retrieve the data as it becomes available. The relay files | |
18 | themselves are files created in a host filesystem, e.g. debugfs, and | |
19 | are associated with the channel buffers using the API described below. | |
20 | ||
21 | The format of the data logged into the channel buffers is completely | |
22 | up to the kernel client; the relay interface does however provide | |
23 | hooks which allow kernel clients to impose some structure on the | |
24 | buffer data. The relay interface doesn't implement any form of data | |
25 | filtering - this also is left to the kernel client. The purpose is to | |
26 | keep things as simple as possible. | |
27 | ||
28 | This document provides an overview of the relay interface API. The | |
29 | details of the function parameters are documented along with the | |
30 | functions in the relay interface code - please see that for details. | |
31 | ||
32 | Semantics | |
33 | ========= | |
34 | ||
35 | Each relay channel has one buffer per CPU, each buffer has one or more | |
36 | sub-buffers. Messages are written to the first sub-buffer until it is | |
c9f3f2d8 | 37 | too full to contain a new message, in which case it is written to |
e88d78f6 TZ |
38 | the next (if available). Messages are never split across sub-buffers. |
39 | At this point, userspace can be notified so it empties the first | |
40 | sub-buffer, while the kernel continues writing to the next. | |
41 | ||
42 | When notified that a sub-buffer is full, the kernel knows how many | |
43 | bytes of it are padding i.e. unused space occurring because a complete | |
44 | message couldn't fit into a sub-buffer. Userspace can use this | |
45 | knowledge to copy only valid data. | |
46 | ||
47 | After copying it, userspace can notify the kernel that a sub-buffer | |
48 | has been consumed. | |
49 | ||
50 | A relay channel can operate in a mode where it will overwrite data not | |
51 | yet collected by userspace, and not wait for it to be consumed. | |
52 | ||
53 | The relay channel itself does not provide for communication of such | |
54 | data between userspace and kernel, allowing the kernel side to remain | |
55 | simple and not impose a single interface on userspace. It does | |
56 | provide a set of examples and a separate helper though, described | |
57 | below. | |
58 | ||
59 | The read() interface both removes padding and internally consumes the | |
60 | read sub-buffers; thus in cases where read(2) is being used to drain | |
61 | the channel buffers, special-purpose communication between kernel and | |
62 | user isn't necessary for basic operation. | |
63 | ||
64 | One of the major goals of the relay interface is to provide a low | |
65 | overhead mechanism for conveying kernel data to userspace. While the | |
66 | read() interface is easy to use, it's not as efficient as the mmap() | |
67 | approach; the example code attempts to make the tradeoff between the | |
68 | two approaches as small as possible. | |
69 | ||
70 | klog and relay-apps example code | |
71 | ================================ | |
72 | ||
73 | The relay interface itself is ready to use, but to make things easier, | |
74 | a couple simple utility functions and a set of examples are provided. | |
75 | ||
76 | The relay-apps example tarball, available on the relay sourceforge | |
77 | site, contains a set of self-contained examples, each consisting of a | |
78 | pair of .c files containing boilerplate code for each of the user and | |
79 | kernel sides of a relay application. When combined these two sets of | |
80 | boilerplate code provide glue to easily stream data to disk, without | |
81 | having to bother with mundane housekeeping chores. | |
82 | ||
83 | The 'klog debugging functions' patch (klog.patch in the relay-apps | |
84 | tarball) provides a couple of high-level logging functions to the | |
85 | kernel which allow writing formatted text or raw data to a channel, | |
86 | regardless of whether a channel to write into exists or not, or even | |
87 | whether the relay interface is compiled into the kernel or not. These | |
88 | functions allow you to put unconditional 'trace' statements anywhere | |
89 | in the kernel or kernel modules; only when there is a 'klog handler' | |
90 | registered will data actually be logged (see the klog and kleak | |
91 | examples for details). | |
92 | ||
93 | It is of course possible to use the relay interface from scratch, | |
94 | i.e. without using any of the relay-apps example code or klog, but | |
95 | you'll have to implement communication between userspace and kernel, | |
96 | allowing both to convey the state of buffers (full, empty, amount of | |
97 | padding). The read() interface both removes padding and internally | |
98 | consumes the read sub-buffers; thus in cases where read(2) is being | |
99 | used to drain the channel buffers, special-purpose communication | |
100 | between kernel and user isn't necessary for basic operation. Things | |
101 | such as buffer-full conditions would still need to be communicated via | |
102 | some channel though. | |
103 | ||
104 | klog and the relay-apps examples can be found in the relay-apps | |
105 | tarball on http://relayfs.sourceforge.net | |
106 | ||
107 | The relay interface user space API | |
108 | ================================== | |
109 | ||
110 | The relay interface implements basic file operations for user space | |
111 | access to relay channel buffer data. Here are the file operations | |
112 | that are available and some comments regarding their behavior: | |
113 | ||
56e6d5c0 | 114 | =========== ============================================================ |
e88d78f6 TZ |
115 | open() enables user to open an _existing_ channel buffer. |
116 | ||
117 | mmap() results in channel buffer being mapped into the caller's | |
118 | memory space. Note that you can't do a partial mmap - you | |
119 | must map the entire file, which is NRBUF * SUBBUFSIZE. | |
120 | ||
121 | read() read the contents of a channel buffer. The bytes read are | |
122 | 'consumed' by the reader, i.e. they won't be available | |
123 | again to subsequent reads. If the channel is being used | |
124 | in no-overwrite mode (the default), it can be read at any | |
125 | time even if there's an active kernel writer. If the | |
126 | channel is being used in overwrite mode and there are | |
127 | active channel writers, results may be unpredictable - | |
128 | users should make sure that all logging to the channel has | |
129 | ended before using read() with overwrite mode. Sub-buffer | |
130 | padding is automatically removed and will not be seen by | |
131 | the reader. | |
132 | ||
133 | sendfile() transfer data from a channel buffer to an output file | |
134 | descriptor. Sub-buffer padding is automatically removed | |
135 | and will not be seen by the reader. | |
136 | ||
137 | poll() POLLIN/POLLRDNORM/POLLERR supported. User applications are | |
138 | notified when sub-buffer boundaries are crossed. | |
139 | ||
140 | close() decrements the channel buffer's refcount. When the refcount | |
141 | reaches 0, i.e. when no process or kernel client has the | |
142 | buffer open, the channel buffer is freed. | |
56e6d5c0 | 143 | =========== ============================================================ |
e88d78f6 TZ |
144 | |
145 | In order for a user application to make use of relay files, the | |
56e6d5c0 | 146 | host filesystem must be mounted. For example:: |
e88d78f6 | 147 | |
254012fd | 148 | mount -t debugfs debugfs /sys/kernel/debug |
e88d78f6 | 149 | |
56e6d5c0 MCC |
150 | .. Note:: |
151 | ||
152 | the host filesystem doesn't need to be mounted for kernel | |
e88d78f6 TZ |
153 | clients to create or use channels - it only needs to be |
154 | mounted when user space applications need access to the buffer | |
155 | data. | |
156 | ||
157 | ||
158 | The relay interface kernel API | |
159 | ============================== | |
160 | ||
161 | Here's a summary of the API the relay interface provides to in-kernel clients: | |
162 | ||
163 | TBD(curr. line MT:/API/) | |
56e6d5c0 | 164 | channel management functions:: |
e88d78f6 TZ |
165 | |
166 | relay_open(base_filename, parent, subbuf_size, n_subbufs, | |
23c88752 | 167 | callbacks, private_data) |
e88d78f6 TZ |
168 | relay_close(chan) |
169 | relay_flush(chan) | |
170 | relay_reset(chan) | |
171 | ||
56e6d5c0 | 172 | channel management typically called on instigation of userspace:: |
e88d78f6 TZ |
173 | |
174 | relay_subbufs_consumed(chan, cpu, subbufs_consumed) | |
175 | ||
56e6d5c0 | 176 | write functions:: |
e88d78f6 TZ |
177 | |
178 | relay_write(chan, data, length) | |
179 | __relay_write(chan, data, length) | |
180 | relay_reserve(chan, length) | |
181 | ||
56e6d5c0 | 182 | callbacks:: |
e88d78f6 TZ |
183 | |
184 | subbuf_start(buf, subbuf, prev_subbuf, prev_padding) | |
185 | buf_mapped(buf, filp) | |
186 | buf_unmapped(buf, filp) | |
187 | create_buf_file(filename, parent, mode, buf, is_global) | |
188 | remove_buf_file(dentry) | |
189 | ||
56e6d5c0 | 190 | helper functions:: |
e88d78f6 TZ |
191 | |
192 | relay_buf_full(buf) | |
193 | subbuf_start_reserve(buf, length) | |
194 | ||
195 | ||
196 | Creating a channel | |
197 | ------------------ | |
198 | ||
199 | relay_open() is used to create a channel, along with its per-cpu | |
200 | channel buffers. Each channel buffer will have an associated file | |
201 | created for it in the host filesystem, which can be and mmapped or | |
202 | read from in user space. The files are named basename0...basenameN-1 | |
203 | where N is the number of online cpus, and by default will be created | |
204 | in the root of the filesystem (if the parent param is NULL). If you | |
205 | want a directory structure to contain your relay files, you should | |
206 | create it using the host filesystem's directory creation function, | |
207 | e.g. debugfs_create_dir(), and pass the parent directory to | |
208 | relay_open(). Users are responsible for cleaning up any directory | |
209 | structure they create, when the channel is closed - again the host | |
210 | filesystem's directory removal functions should be used for that, | |
211 | e.g. debugfs_remove(). | |
212 | ||
213 | In order for a channel to be created and the host filesystem's files | |
214 | associated with its channel buffers, the user must provide definitions | |
215 | for two callback functions, create_buf_file() and remove_buf_file(). | |
216 | create_buf_file() is called once for each per-cpu buffer from | |
217 | relay_open() and allows the user to create the file which will be used | |
218 | to represent the corresponding channel buffer. The callback should | |
219 | return the dentry of the file created to represent the channel buffer. | |
220 | remove_buf_file() must also be defined; it's responsible for deleting | |
221 | the file(s) created in create_buf_file() and is called during | |
222 | relay_close(). | |
223 | ||
224 | Here are some typical definitions for these callbacks, in this case | |
56e6d5c0 MCC |
225 | using debugfs:: |
226 | ||
227 | /* | |
228 | * create_buf_file() callback. Creates relay file in debugfs. | |
229 | */ | |
230 | static struct dentry *create_buf_file_handler(const char *filename, | |
231 | struct dentry *parent, | |
232 | umode_t mode, | |
233 | struct rchan_buf *buf, | |
234 | int *is_global) | |
235 | { | |
236 | return debugfs_create_file(filename, mode, parent, buf, | |
237 | &relay_file_operations); | |
238 | } | |
239 | ||
240 | /* | |
241 | * remove_buf_file() callback. Removes relay file from debugfs. | |
242 | */ | |
243 | static int remove_buf_file_handler(struct dentry *dentry) | |
244 | { | |
245 | debugfs_remove(dentry); | |
246 | ||
247 | return 0; | |
248 | } | |
249 | ||
250 | /* | |
251 | * relay interface callbacks | |
252 | */ | |
253 | static struct rchan_callbacks relay_callbacks = | |
254 | { | |
255 | .create_buf_file = create_buf_file_handler, | |
256 | .remove_buf_file = remove_buf_file_handler, | |
257 | }; | |
258 | ||
259 | And an example relay_open() invocation using them:: | |
e88d78f6 | 260 | |
23c88752 | 261 | chan = relay_open("cpu", NULL, SUBBUF_SIZE, N_SUBBUFS, &relay_callbacks, NULL); |
e88d78f6 TZ |
262 | |
263 | If the create_buf_file() callback fails, or isn't defined, channel | |
264 | creation and thus relay_open() will fail. | |
265 | ||
266 | The total size of each per-cpu buffer is calculated by multiplying the | |
267 | number of sub-buffers by the sub-buffer size passed into relay_open(). | |
268 | The idea behind sub-buffers is that they're basically an extension of | |
269 | double-buffering to N buffers, and they also allow applications to | |
270 | easily implement random-access-on-buffer-boundary schemes, which can | |
271 | be important for some high-volume applications. The number and size | |
272 | of sub-buffers is completely dependent on the application and even for | |
273 | the same application, different conditions will warrant different | |
274 | values for these parameters at different times. Typically, the right | |
275 | values to use are best decided after some experimentation; in general, | |
276 | though, it's safe to assume that having only 1 sub-buffer is a bad | |
277 | idea - you're guaranteed to either overwrite data or lose events | |
278 | depending on the channel mode being used. | |
279 | ||
280 | The create_buf_file() implementation can also be defined in such a way | |
281 | as to allow the creation of a single 'global' buffer instead of the | |
282 | default per-cpu set. This can be useful for applications interested | |
283 | mainly in seeing the relative ordering of system-wide events without | |
284 | the need to bother with saving explicit timestamps for the purpose of | |
285 | merging/sorting per-cpu files in a postprocessing step. | |
286 | ||
287 | To have relay_open() create a global buffer, the create_buf_file() | |
288 | implementation should set the value of the is_global outparam to a | |
289 | non-zero value in addition to creating the file that will be used to | |
290 | represent the single buffer. In the case of a global buffer, | |
291 | create_buf_file() and remove_buf_file() will be called only once. The | |
292 | normal channel-writing functions, e.g. relay_write(), can still be | |
293 | used - writes from any cpu will transparently end up in the global | |
294 | buffer - but since it is a global buffer, callers should make sure | |
295 | they use the proper locking for such a buffer, either by wrapping | |
296 | writes in a spinlock, or by copying a write function from relay.h and | |
297 | creating a local version that internally does the proper locking. | |
298 | ||
23c88752 MD |
299 | The private_data passed into relay_open() allows clients to associate |
300 | user-defined data with a channel, and is immediately available | |
301 | (including in create_buf_file()) via chan->private_data or | |
302 | buf->chan->private_data. | |
303 | ||
20d8b67c EGM |
304 | Buffer-only channels |
305 | -------------------- | |
306 | ||
307 | These channels have no files associated and can be created with | |
308 | relay_open(NULL, NULL, ...). Such channels are useful in scenarios such | |
309 | as when doing early tracing in the kernel, before the VFS is up. In these | |
310 | cases, one may open a buffer-only channel and then call | |
311 | relay_late_setup_files() when the kernel is ready to handle files, | |
312 | to expose the buffered data to the userspace. | |
313 | ||
e88d78f6 TZ |
314 | Channel 'modes' |
315 | --------------- | |
316 | ||
317 | relay channels can be used in either of two modes - 'overwrite' or | |
318 | 'no-overwrite'. The mode is entirely determined by the implementation | |
319 | of the subbuf_start() callback, as described below. The default if no | |
320 | subbuf_start() callback is defined is 'no-overwrite' mode. If the | |
321 | default mode suits your needs, and you plan to use the read() | |
322 | interface to retrieve channel data, you can ignore the details of this | |
323 | section, as it pertains mainly to mmap() implementations. | |
324 | ||
325 | In 'overwrite' mode, also known as 'flight recorder' mode, writes | |
326 | continuously cycle around the buffer and will never fail, but will | |
327 | unconditionally overwrite old data regardless of whether it's actually | |
328 | been consumed. In no-overwrite mode, writes will fail, i.e. data will | |
329 | be lost, if the number of unconsumed sub-buffers equals the total | |
330 | number of sub-buffers in the channel. It should be clear that if | |
331 | there is no consumer or if the consumer can't consume sub-buffers fast | |
332 | enough, data will be lost in either case; the only difference is | |
333 | whether data is lost from the beginning or the end of a buffer. | |
334 | ||
335 | As explained above, a relay channel is made of up one or more | |
336 | per-cpu channel buffers, each implemented as a circular buffer | |
337 | subdivided into one or more sub-buffers. Messages are written into | |
338 | the current sub-buffer of the channel's current per-cpu buffer via the | |
339 | write functions described below. Whenever a message can't fit into | |
340 | the current sub-buffer, because there's no room left for it, the | |
341 | client is notified via the subbuf_start() callback that a switch to a | |
342 | new sub-buffer is about to occur. The client uses this callback to 1) | |
343 | initialize the next sub-buffer if appropriate 2) finalize the previous | |
344 | sub-buffer if appropriate and 3) return a boolean value indicating | |
345 | whether or not to actually move on to the next sub-buffer. | |
346 | ||
347 | To implement 'no-overwrite' mode, the userspace client would provide | |
348 | an implementation of the subbuf_start() callback something like the | |
56e6d5c0 | 349 | following:: |
e88d78f6 | 350 | |
56e6d5c0 MCC |
351 | static int subbuf_start(struct rchan_buf *buf, |
352 | void *subbuf, | |
353 | void *prev_subbuf, | |
354 | unsigned int prev_padding) | |
355 | { | |
356 | if (prev_subbuf) | |
357 | *((unsigned *)prev_subbuf) = prev_padding; | |
e88d78f6 | 358 | |
56e6d5c0 MCC |
359 | if (relay_buf_full(buf)) |
360 | return 0; | |
e88d78f6 | 361 | |
56e6d5c0 | 362 | subbuf_start_reserve(buf, sizeof(unsigned int)); |
e88d78f6 | 363 | |
56e6d5c0 MCC |
364 | return 1; |
365 | } | |
e88d78f6 TZ |
366 | |
367 | If the current buffer is full, i.e. all sub-buffers remain unconsumed, | |
368 | the callback returns 0 to indicate that the buffer switch should not | |
369 | occur yet, i.e. until the consumer has had a chance to read the | |
370 | current set of ready sub-buffers. For the relay_buf_full() function | |
a982ac06 | 371 | to make sense, the consumer is responsible for notifying the relay |
e88d78f6 TZ |
372 | interface when sub-buffers have been consumed via |
373 | relay_subbufs_consumed(). Any subsequent attempts to write into the | |
374 | buffer will again invoke the subbuf_start() callback with the same | |
375 | parameters; only when the consumer has consumed one or more of the | |
376 | ready sub-buffers will relay_buf_full() return 0, in which case the | |
377 | buffer switch can continue. | |
378 | ||
379 | The implementation of the subbuf_start() callback for 'overwrite' mode | |
56e6d5c0 | 380 | would be very similar:: |
e88d78f6 | 381 | |
56e6d5c0 MCC |
382 | static int subbuf_start(struct rchan_buf *buf, |
383 | void *subbuf, | |
384 | void *prev_subbuf, | |
385 | size_t prev_padding) | |
386 | { | |
387 | if (prev_subbuf) | |
388 | *((unsigned *)prev_subbuf) = prev_padding; | |
e88d78f6 | 389 | |
56e6d5c0 | 390 | subbuf_start_reserve(buf, sizeof(unsigned int)); |
e88d78f6 | 391 | |
56e6d5c0 MCC |
392 | return 1; |
393 | } | |
e88d78f6 TZ |
394 | |
395 | In this case, the relay_buf_full() check is meaningless and the | |
396 | callback always returns 1, causing the buffer switch to occur | |
397 | unconditionally. It's also meaningless for the client to use the | |
398 | relay_subbufs_consumed() function in this mode, as it's never | |
399 | consulted. | |
400 | ||
401 | The default subbuf_start() implementation, used if the client doesn't | |
402 | define any callbacks, or doesn't define the subbuf_start() callback, | |
403 | implements the simplest possible 'no-overwrite' mode, i.e. it does | |
404 | nothing but return 0. | |
405 | ||
406 | Header information can be reserved at the beginning of each sub-buffer | |
407 | by calling the subbuf_start_reserve() helper function from within the | |
408 | subbuf_start() callback. This reserved area can be used to store | |
409 | whatever information the client wants. In the example above, room is | |
410 | reserved in each sub-buffer to store the padding count for that | |
411 | sub-buffer. This is filled in for the previous sub-buffer in the | |
412 | subbuf_start() implementation; the padding value for the previous | |
413 | sub-buffer is passed into the subbuf_start() callback along with a | |
414 | pointer to the previous sub-buffer, since the padding value isn't | |
415 | known until a sub-buffer is filled. The subbuf_start() callback is | |
416 | also called for the first sub-buffer when the channel is opened, to | |
417 | give the client a chance to reserve space in it. In this case the | |
418 | previous sub-buffer pointer passed into the callback will be NULL, so | |
419 | the client should check the value of the prev_subbuf pointer before | |
420 | writing into the previous sub-buffer. | |
421 | ||
422 | Writing to a channel | |
423 | -------------------- | |
424 | ||
425 | Kernel clients write data into the current cpu's channel buffer using | |
426 | relay_write() or __relay_write(). relay_write() is the main logging | |
427 | function - it uses local_irqsave() to protect the buffer and should be | |
428 | used if you might be logging from interrupt context. If you know | |
429 | you'll never be logging from interrupt context, you can use | |
430 | __relay_write(), which only disables preemption. These functions | |
431 | don't return a value, so you can't determine whether or not they | |
432 | failed - the assumption is that you wouldn't want to check a return | |
433 | value in the fast logging path anyway, and that they'll always succeed | |
434 | unless the buffer is full and no-overwrite mode is being used, in | |
435 | which case you can detect a failed write in the subbuf_start() | |
436 | callback by calling the relay_buf_full() helper function. | |
437 | ||
438 | relay_reserve() is used to reserve a slot in a channel buffer which | |
439 | can be written to later. This would typically be used in applications | |
440 | that need to write directly into a channel buffer without having to | |
441 | stage data in a temporary buffer beforehand. Because the actual write | |
442 | may not happen immediately after the slot is reserved, applications | |
443 | using relay_reserve() can keep a count of the number of bytes actually | |
444 | written, either in space reserved in the sub-buffers themselves or as | |
445 | a separate array. See the 'reserve' example in the relay-apps tarball | |
446 | at http://relayfs.sourceforge.net for an example of how this can be | |
447 | done. Because the write is under control of the client and is | |
448 | separated from the reserve, relay_reserve() doesn't protect the buffer | |
449 | at all - it's up to the client to provide the appropriate | |
450 | synchronization when using relay_reserve(). | |
451 | ||
452 | Closing a channel | |
453 | ----------------- | |
454 | ||
455 | The client calls relay_close() when it's finished using the channel. | |
456 | The channel and its associated buffers are destroyed when there are no | |
457 | longer any references to any of the channel buffers. relay_flush() | |
458 | forces a sub-buffer switch on all the channel buffers, and can be used | |
459 | to finalize and process the last sub-buffers before the channel is | |
460 | closed. | |
461 | ||
462 | Misc | |
463 | ---- | |
464 | ||
465 | Some applications may want to keep a channel around and re-use it | |
466 | rather than open and close a new channel for each use. relay_reset() | |
467 | can be used for this purpose - it resets a channel to its initial | |
468 | state without reallocating channel buffer memory or destroying | |
469 | existing mappings. It should however only be called when it's safe to | |
470 | do so, i.e. when the channel isn't currently being written to. | |
471 | ||
472 | Finally, there are a couple of utility callbacks that can be used for | |
473 | different purposes. buf_mapped() is called whenever a channel buffer | |
474 | is mmapped from user space and buf_unmapped() is called when it's | |
475 | unmapped. The client can use this notification to trigger actions | |
476 | within the kernel application, such as enabling/disabling logging to | |
477 | the channel. | |
478 | ||
479 | ||
480 | Resources | |
481 | ========= | |
482 | ||
483 | For news, example code, mailing list, etc. see the relay interface homepage: | |
484 | ||
485 | http://relayfs.sourceforge.net | |
486 | ||
487 | ||
488 | Credits | |
489 | ======= | |
490 | ||
491 | The ideas and specs for the relay interface came about as a result of | |
492 | discussions on tracing involving the following: | |
493 | ||
494 | Michel Dagenais <michel.dagenais@polymtl.ca> | |
495 | Richard Moore <richardj_moore@uk.ibm.com> | |
496 | Bob Wisniewski <bob@watson.ibm.com> | |
497 | Karim Yaghmour <karim@opersys.com> | |
498 | Tom Zanussi <zanussi@us.ibm.com> | |
499 | ||
500 | Also thanks to Hubertus Franke for a lot of useful suggestions and bug | |
501 | reports. |