Cleanup: made postcat "record names" output more consistent.
+20030421
+
+ Debugging: added some extra detailed error logging to the
+ pipe-to-command delivery, to help folks with bizarre file
+ truncation problems. File: global/pipe_command.c.
+
+20030424
+
+ Cleanup: readlline() did not terminate the result before
+ complaining about lines starting with whitespace.
+
+ Cleanup: eliminated valid_hostname warning for invalid
+ queue file names. File: global/mail_queue.c.
+
+ Bugfix: lost three lines of code when readying the postcat
+ command for release, which broke postcat -q. File:
+ postcat/postcat.c.
+
+ Bugfix: the Postfix sendmail command applied the message
+ size limit when running as newaliases. The limiting code
+ is now moved to the message enqueuing branch of the code.
+ File: sendmail/sendmail.c.
+
+ Documentation: start of documentation for the algorithm of
+ Patrik Rak's clever queue manager scheduler (nqmgr). Files:
+ conf/sample-scheduler.cf, README_FILES/SCHEDULER_README.
+
Open problems:
Low: smtp-source may block when sending large test messages.
SASL authentication information is not passed on via message headers
or via SMTP. It is no-one's business what username and authentication
-method the poster was using in order to access the mail server.
+method the poster was using in order to access the mail server. The
+people who need to know can find the information in the maillog file.
When sending mail, Postfix looks up the server hostname or destination
domain (the address remote part) in a table, and if a username/password
--- /dev/null
+What this file is about
+=======================
+
+This is the beginning of documentation for the clever queue manager
+scheduling algorithm by Patrik Rak. For too much time, this code
+has been made available in the nqmgr queue manager as an optional
+module.
+
+Why Postfix ships two queue managers
+====================================
+
+The old Postfix scheduler had several limitations due to unfortunate
+choices in its design.
+
+1 - Round-robin selection by destination for mail that is delivered
+ via the same message delivery transport. That strategy broke
+ down when one single destination (say, inbound mail) had to
+ compete with multiple other destinations (say, outbound mail).
+ The poor suffering destination would be selected only
+ 1/number-of-destinations of the time, even when it had more
+ mail than other destinations, and thus mail could be delayed.
+
+ Victor Duchovni found a workaround: use different message
+ delivery transports, and thus avoid the starvation problem.
+ The Patrik Rak scheduler solves this problem by using FIFO
+ selection.
+
+2 - A second limitation of the old Postfix scheduler was that
+ delivery of bulk mail would block all other deliveries, causing
+ large delays. Patrik Rak's scheduler allows mail with fewer
+ recipients to slip past bulk mail in an elegant manner.
+
+If this newer scheduler is so much better, why does Postfix still
+ship the old one? The problem is that there isn't a whole lot of
+documentation on how Patrik's code works, so that no-one except
+Patrik understands how it works.
+
+This document is the start of something that will help to clarify
+things. Once enough documentation exists we can end the embarassing
+situation of shipping two queue managers.
+
+How the queue manager scheduler works
+=====================================
+
+[The following text is from Patrik Rak and should be read together
+with the sample-scheduler.cf file]
+
+From user's point of view, qmgr and nqmgr are both the same, except
+for how next message is chosen when delivery agent becomes available.
+You already know that qmgr uses round-robin by destination while
+nqmgr uses simple FIFO, except for some preemptive magic. The
+[sample-scheduler.cf file] documents all the knobs the user can
+use to control this preemptive magic - there is nothing else to
+the preemption than the quite simple conditions described below.
+
+As for programmer-level documentation, this will have to be extracted
+from all those emails we have exchanged with Wietse [rats! I hoped
+that Patrik would do the work for me -- Wietse] But I think there
+are no missing bits which we have not mentioned in our conversations.
+
+However, even from programmer's point of view, there is nothing
+more to add to the message scheduling idea itself. There are few
+things which make it look more complicated than it is, but the
+algorithm is the same as the user percieves it. The summary of the
+changes from the user's view:
+
+1) Simplification of terms for users: The user knows about messages
+and recipients. The program itself works with jobs (one message is
+split among several jobs, one per each transport needed to deliver
+the message) and queue entries (each entry may group several
+recipients for same destination). Then there is the peer structure
+introduced by nqmgr which is simply per-job analog of the queue
+structure.
+
+2) Dealing with concurrency limits: The actual implementation is
+complicated by the fact that the messages (resp. jobs) may not be
+delivered in the exactly scheduled order because of the concurrency
+limits. It is necessary to skip some "blocker" jobs when the
+concurrency limit is reached and get back to them again when the
+limit permits.
+
+3) Dealing with resource limits: The actual implementation is
+complicated by the fact that not all recipients may be read in-core.
+Therefore each message has some recipients in-core and some may
+remain on-file. This means that a) the preemptive algorithm needs
+to work with recipient count estimates instead of exact counts, b)
+there is extra code which needs to manipulate the per-transport
+pool of recipients which may be read in-core at the same time, and
+c) there is extra code which needs to be able to read recipients
+into core in batches and which is triggered at appropriate moments.
+
+4) Doing things efficiently: All important things I am aware of
+are done in the minimum time possible (either directly or at least
+when ammortized complexity is used), but to choose which job is
+the best candidate for preempting the current job requires linear
+search of up to all transport jobs (the worst theoretical case -
+the reality is much better). As this is done every time the next
+queue entry to be delivered is about to be chosen, it seemed
+reasonable to add cache which minimizes the overhead. Maintenance
+of this candidate cache slightly obfuscates things.
+
+The points 2 and 3 are those which made the implementation (look)
+complicated and were the real coding work, but I believe that to
+understand the scheduling algorithm itself (which was the real
+thinking work) is fairly easy.
maps, and the mailbox location map can specify either mailbox or
maildir delivery (controlled by trailing slash on mailbox name).
-The agent does not support user+foo address extensions, aliases or
-.forward files (use the virtual table instead), and therefore
-doesn't support file or program aliases. This choice was made to
-simplify and streamline the code (it allowed me to dispense with
-70% of local's code - mostly the bits that are a security headache)
-- if you need this functionality, this agent isn't for you.
+The agent allows but ignores user+foo address extensions, does not
+support aliases or .forward files (use the virtual table instead),
+and therefore doesn't support file or program aliases. This choice
+was made to simplify and streamline the code (it allowed me to
+dispense with 70% of local's code - mostly the bits that are a
+security headache) - if you need this functionality, this agent
+isn't for you.
It also doesn't support writing to a common spool as root and then
chowning the mailbox to the user - I felt this functionality didn't
==============================================================
This example does not use the Postfix local delivery agent at all.
-With this configuration Postfix does no user+foo address extension,
-no alias expansion, no .forward file expansion, and no lookups of
-recipients in /etc/passwd.
+With this configuration Postfix does no alias expansion, no .forward
+file expansion, no lookups of recipients in /etc/passwd, and allows
+but ignores user+foo address extensions.
Instead of "hash" specify "dbm" or "btree", depending on your system
type. The command "postconf -m" displays possible lookup table
$sample_directory/sample-relocated.cf:f:root:-:644
$sample_directory/sample-resource.cf:f:root:-:644
$sample_directory/sample-rewrite.cf:f:root:-:644
+$sample_directory/sample-scheduler.cf:f:root:-:644
$sample_directory/sample-smtp.cf:f:root:-:644
$sample_directory/sample-smtpd.cf:f:root:-:644
$sample_directory/sample-transport.cf:f:root:-:644
$readme_directory/RELEASE_NOTES:f:root:-:644
$readme_directory/RESTRICTION_CLASS_README:f:root:-:644
$readme_directory/SASL_README:f:root:-:644
+$readme_directory/SCHEDULER_README:f:root:-:644
$readme_directory/ULTRIX_README:f:root:-:644
$readme_directory/UUCP_README:f:root:-:644
$readme_directory/VERP_README:f:root:-:644
# The qmgr_message_active_limit parameter limits the number of
# messages in the active queue.
#
-qmgr_message_active_limit = 1000
+qmgr_message_active_limit = 10000
# The qmgr_message_recipient_limit parameter limits the number of
-# in-memory recipients. This parameter also limits the size of the
-# short-term, in-memory destination status cache.
+# in-memory recipients. This limit is used before any of the message
+# recipients have been read and the message has not been assigned
+# to any transports yet. See also qmgr_message_recipient_minimum.
#
-qmgr_message_recipient_limit = 1000
+# This parameter also limits the size of the short-term, in-memory
+# destination status cache.
+#
+qmgr_message_recipient_limit = 10000
+
+# (nqmgr only)
+# The default_recipient_limit parameter is the default value for
+# per-transport limit imposed on the number of in-memory recipients.
+# These limits take priority over the global qmgr_message_recipient_limit
+# after the message has been assigned to the respective transports.
+# See also default_extra_recipient_limit and qmgr_message_recipient_minimum.
+#
+default_recipient_limit = 10000
+
+# (nqmgr only)
+# The default_extra_recipient_limit parameter is the default value for
+# extra per-transport limit imposed on the number of in-memory recipients.
+# This extra recipient space is reserved for the cases when the scheduler
+# preempts one message with another and suddenly needs some extra recipients
+# slots for the chosen message in order to avoid performance degradation.
+#
+default_extra_recipient_limit = 1000
+
+# (nqmgr only)
+# The qmgr_message_recipient_minimum parameter specifies the minimum
+# amount of in-memory recipients for any message. This takes
+# priority over any other in-memory recipient limits (i.e., global
+# qmgr_message_recipient_limit and per transport XXX_recipient_limit)
+# if necessary. The minimum value allowed for this parameter is 1.
+#
+qmgr_message_recipient_minimum = 10
# The queue_minfree parameter specifies the minimal amount of free
# space in bytes in the queue file system. This is currently used by
--- /dev/null
+# DO NOT EDIT THIS FILE. EDIT THE MAIN.CF FILE INSTEAD. THE STUFF
+# HERE JUST SERVES AS AN EXAMPLE.
+#
+# This file contains example settings of Postfix queue manager
+# scheduler parameters. It is specific to the nqmgr queue manager.
+#
+# Controls that are specific to message delivery transports are
+# described in the respective sample-transportname.cf file.
+
+# The default_delivery_slot_cost parameter controls how often
+# is the scheduler allowed to preempt one message with another.
+#
+# Each transport maintains so-called "available delivery slot counter" for
+# each message. One message can be preempted by another one when the other
+# message can be delivered using no more delivery slots (i.e., invocations
+# of delivery agents) than the current message counter has accumulated (or
+# will eventually accumulate - see about slot loans below). This parameter
+# controls how often is the counter incremented - it happens after each
+# default_delivery_slot_cost recipients have been delivered.
+#
+# The cost of 0 is used to disable the preempting scheduling completely. The
+# minimum value the scheduling algorithm can use is 2 - use it if you want
+# to maximize the message throughput rate. Although there is no maximum, it
+# doesn't make much sense to use values above say 50.
+#
+# The only reason why the value of 2 is not the default is the way this
+# parameter affects the delivery of mailing-list mail. In the worst case,
+# their delivery can take somewhere between (cost+1/cost) and (cost/cost-1)
+# times more than if the preemptive scheduler was disabled. The default
+# value of 5 turns out to provide reasonable message response times while
+# making sure the mailing-list deliveries are not extended by more than
+# 20-25 percent even in the worst case.
+#
+# default_delivery_slot_cost = 0
+# default_delivery_slot_cost = 2
+default_delivery_slot_cost = 5
+
+# The default_delivery_slot_discount and default_delivery_slot_loan
+# parameters allow fine-tuning the exact moment when the message preemption
+# happens. It turns out that most of the time it would be wasteful to delay
+# the message preemption until the delivery slot counter accumulates the
+# full amount of delivery slots needed for the preemption to happen. On the
+# other hand, it would not be not very smart to do every preemption
+# immediately either, because it could use all slots that some other urgent
+# message may need just seconds later.
+#
+# The discount (resp. loan) parameter specifies how many percent (resp. how
+# many slots) of the total amount one "gets in advance". If both discount
+# and loan are 0, the preemption can only happen after the full amount has
+# accumulated. With discount of 100, the preemption would happen immediately.
+# The default settings allow the preemption happen after about half of the
+# total amount has accumulated while further reducing the delay for messages
+# with very little recipients.
+#
+default_delivery_slot_discount = 50
+default_delivery_slot_loan = 3
+
+# The default_minimum_delivery_slots parameter controls how many recipients
+# a message must have in order to invoke the scheduling algorithm at all.
+# Messages which would never accumulate at least this many delivery slots
+# (subject to slot cost parameter as well) are never preempted.
+#
+default_minimum_delivery_slots = 3
ular Expressions. The file format is
described in <a href="pcre_table.5.html"><b>pcre</b><i>_</i><b>table</b>(5)</a>.
+ <b>pgsql</b> (read-only)
+ Perform lookups using the PostgreSQL proto-
+ col. This is described in a PGSQL_README
+ file.
+
<b>proxy</b> (read-only)
A lookup table that is implemented via the
Postfix <a href="proxymap.8.html"><b>proxymap</b>(8)</a> service. The table name
.IP "\fBpcre\fR (read-only)"
A lookup table based on Perl Compatible Regular Expressions. The
file format is described in \fBpcre_table\fR(5).
+.IP "\fBpgsql\fR (read-only)"
+Perform lookups using the PostgreSQL protocol. This is described
+in a PGSQL_README file.
.IP "\fBproxy\fR (read-only)"
A lookup table that is implemented via the Postfix
\fBproxymap\fR(8) service. The table name syntax is
/*
* OK if in valid hostname form.
*/
- return (valid_hostname(queue_id, DO_GRIPE));
+ return (valid_hostname(queue_id, DONT_GRIPE));
}
/* mail_queue_enter - make mail queue entry with locally-unique name */
* Patches change the patchlevel and the release date. Snapshots change the
* release date only, unless they include the same bugfix as a patch release.
*/
-#define MAIL_RELEASE_DATE "20030419"
+#define MAIL_RELEASE_DATE "20030424"
#define VAR_MAIL_VERSION "mail_version"
#define DEF_MAIL_VERSION "2.0.9-" MAIL_RELEASE_DATE
*/
if (write_wait(fd, maxtime) < 0) {
if (pipe_command_timeout == 0) {
- if (msg_verbose)
- msg_info("%s: time limit exceeded", myname);
+ msg_warn("%s: write time limit exceeded", myname);
pipe_command_timeout = 1;
}
return (0);
*/
if (read_wait(fd, maxtime) < 0) {
if (pipe_command_timeout == 0) {
- if (msg_verbose)
- msg_info("%s: time limit exceeded", myname);
+ msg_warn("%s: read time limit exceeded", myname);
pipe_command_timeout = 1;
}
return (0);
*/
if ((n = timed_waitpid(pid, statusp, 0, maxtime)) < 0 && errno == ETIMEDOUT) {
if (pipe_command_timeout == 0) {
- if (msg_verbose)
- msg_info("%s: time limit exceeded", myname);
+ msg_warn("%s: child wait time limit exceeded", myname);
pipe_command_timeout = 1;
}
kill_command(pid, sig, kill_uid, kill_gid);
int log_len;
pid_t pid;
int write_status;
+ int write_errno;
WAIT_STATUS_T wait_status;
int cmd_in_pipe[2];
int cmd_out_pipe[2];
args.delivered, src,
cmd_in_stream, args.flags,
args.eol, DONT_CARE_WHY);
+ write_errno = errno;
/*
* Capture a limited amount of command output, for inclusion in a
}
} else if (write_status & MAIL_COPY_STAT_CORRUPT) {
return (PIPE_STAT_CORRUPT);
- } else if (write_status && errno != EPIPE) {
- vstring_sprintf(why, "Command failed: %m: \"%s\"", args.command);
+ } else if (write_status && write_errno != EPIPE) {
+ errno = write_errno;
+ vstring_sprintf(why, "Command failed due to %s: %m: \"%s\"",
+ (write_status & MAIL_COPY_STAT_READ) ? "delivery read error" :
+ (write_status & MAIL_COPY_STAT_WRITE) ? "delivery write error" :
+ "some delivery error", args.command);
return (PIPE_STAT_DEFER);
} else {
return (PIPE_STAT_OK);
break;
}
prev_type = rec_type;
+
+ /*
+ * In case the next record is broken.
+ */
+ vstream_fflush(VSTREAM_OUT);
}
- vstream_fflush(VSTREAM_OUT);
}
/* usage - explain and terminate */
if (chdir(var_queue_dir))
msg_fatal("chdir %s: %m", var_queue_dir);
while (optind < argc) {
+ if (!mail_queue_id_ok(argv[optind]))
+ msg_fatal("bad mail queue ID: %s", argv[optind]);
for (fp = 0, tries = 0; fp == 0 && tries < 2; tries++)
for (cpp = queue_names; fp == 0 && *cpp != 0; cpp++)
fp = mail_queue_open(*cpp, argv[optind], O_RDONLY, 0);
if (fp == 0)
msg_fatal("open queue file %s: %m", argv[optind]);
+ postcat(fp, buffer);
+ if (vstream_fclose(fp))
+ msg_warn("close %s: %m", argv[optind]);
optind++;
}
}
/* .IP "\fBpcre\fR (read-only)"
/* A lookup table based on Perl Compatible Regular Expressions. The
/* file format is described in \fBpcre_table\fR(5).
+/* .IP "\fBpgsql\fR (read-only)"
+/* Perform lookups using the PostgreSQL protocol. This is described
+/* in a PGSQL_README file.
/* .IP "\fBproxy\fR (read-only)"
/* A lookup table that is implemented via the Postfix
/* \fBproxymap\fR(8) service. The table name syntax is
*/
buf = vstring_alloc(100);
+ /*
+ * Stop run-away process accidents by limiting the queue file size. This
+ * is not a defense against DOS attack.
+ */
+ if (var_message_limit > 0 && get_file_limit() > var_message_limit)
+ set_file_limit((off_t) var_message_limit);
+
/*
* The sender name is provided by the user. In principle, the mail pickup
* service could deduce the sender name from queue file ownership, but:
if (chdir(var_queue_dir))
msg_fatal_status(EX_UNAVAILABLE, "chdir %s: %m", var_queue_dir);
- /*
- * Stop run-away process accidents by limiting the queue file size. This
- * is not a defense against DOS attack.
- */
- if (var_message_limit > 0 && get_file_limit() > var_message_limit)
- set_file_limit((off_t) var_message_limit);
-
signal(SIGPIPE, SIG_IGN);
/*
case DEL_RCPT_STAT_TODO:
case DEL_RCPT_STAT_DEFER:
DEFER_IF_PERMIT3(state, MAIL_ERROR_POLICY,
- "450 <%s>: %s rejected: unverified address: %s",
+ "450 <%s>: %s rejected: unverified address: %.250s",
reply_name, reply_class, STR(why));
rqst_status = SMTPD_CHECK_DUNNO;
break;
smtpd_delay_reject 0
mynetworks 127.0.0.0/8,168.100.189.0/28
relay_domains porcupine.org
-maps_rbl_domains socks.relays.osirusoft.com
+maps_rbl_domains relays.mail-abuse.org
#
# Test the client restrictions.
#
smtpd_delay_reject 0
mynetworks 127.0.0.0/8,168.100.189.0/28
relay_domains porcupine.org
-maps_rbl_domains socks.relays.osirusoft.com
+maps_rbl_domains relays.mail-abuse.org
#
# Test the client restrictions.
#
OK
>>> relay_domains porcupine.org
OK
->>> maps_rbl_domains socks.relays.osirusoft.com
+>>> maps_rbl_domains relays.mail-abuse.org
OK
>>> #
>>> # Test the client restrictions.
./smtpd_check: warning: restriction reject_maps_rbl is going away. Please use reject_rbl_client <domain> instead
OK
>>> client foo 127.0.0.2
-./smtpd_check: <queue id>: reject: CONNECT from foo[127.0.0.2]: 554 Service unavailable; Client host [127.0.0.2] blocked using socks.relays.osirusoft.com; Proxy List Test; from=<foo@friend.bad.domain> proto=SMTP helo=<123.123.123.123>
-554 Service unavailable; Client host [127.0.0.2] blocked using socks.relays.osirusoft.com; Proxy List Test
+./smtpd_check: <queue id>: reject: CONNECT from foo[127.0.0.2]: 554 Service unavailable; Client host [127.0.0.2] blocked using relays.mail-abuse.org; from=<foo@friend.bad.domain> proto=SMTP helo=<123.123.123.123>
+554 Service unavailable; Client host [127.0.0.2] blocked using relays.mail-abuse.org
>>> #
>>> # Hybrids
>>> #
OK
>>> relay_domains porcupine.org
OK
->>> maps_rbl_domains socks.relays.osirusoft.com
+>>> maps_rbl_domains relays.mail-abuse.org
OK
>>> #
>>> # Test the client restrictions.
./smtpd_check: warning: restriction reject_maps_rbl is going away. Please use reject_rbl_client <domain> instead
OK
>>> client foo 127.0.0.2
-./smtpd_check: <queue id>: reject: CONNECT from foo[127.0.0.2]: 554 Service unavailable; Client host [127.0.0.2] blocked using socks.relays.osirusoft.com; Proxy List Test; from=<foo@friend.bad.domain> proto=SMTP helo=<friend.bad.domain>
-554 Service unavailable; Client host [127.0.0.2] blocked using socks.relays.osirusoft.com; Proxy List Test
+./smtpd_check: <queue id>: reject: CONNECT from foo[127.0.0.2]: 554 Service unavailable; Client host [127.0.0.2] blocked using relays.mail-abuse.org; from=<foo@friend.bad.domain> proto=SMTP helo=<friend.bad.domain>
+554 Service unavailable; Client host [127.0.0.2] blocked using relays.mail-abuse.org
>>> #
>>> # unknown sender/recipient domain
>>> #
smtpd_delay_reject 0
mynetworks 127.0.0.0/8,168.100.189.0/28
relay_domains porcupine.org
-maps_rbl_domains socks.relays.osirusoft.com
+maps_rbl_domains relays.mail-abuse.org
rbl_reply_maps hash:smtpd_check_access
helo foobar
#
client foo 127.0.0.2
rcpt rname@rdomain
#
-recipient_restrictions reject_rbl_client,socks.relays.osirusoft.com
+recipient_restrictions reject_rbl_client,relays.mail-abuse.org
client spike.porcupine.org 168.100.189.2
rcpt rname@rdomain
client foo 127.0.0.2
OK
>>> relay_domains porcupine.org
OK
->>> maps_rbl_domains socks.relays.osirusoft.com
+>>> maps_rbl_domains relays.mail-abuse.org
OK
>>> rbl_reply_maps hash:smtpd_check_access
OK
>>> client foo 127.0.0.2
OK
>>> rcpt rname@rdomain
-./smtpd_check: <queue id>: reject: RCPT from foo[127.0.0.2]: 554 Service unavailable; Client host [127.0.0.2] blocked using socks.relays.osirusoft.com; Proxy List Test; from=<sname@sdomain> to=<rname@rdomain> proto=SMTP helo=<foobar>
-554 Service unavailable; Client host [127.0.0.2] blocked using socks.relays.osirusoft.com; Proxy List Test
+./smtpd_check: <queue id>: reject: RCPT from foo[127.0.0.2]: 554 Service unavailable; Client host [127.0.0.2] blocked using relays.mail-abuse.org; from=<sname@sdomain> to=<rname@rdomain> proto=SMTP helo=<foobar>
+554 Service unavailable; Client host [127.0.0.2] blocked using relays.mail-abuse.org
>>> #
->>> recipient_restrictions reject_rbl_client,socks.relays.osirusoft.com
+>>> recipient_restrictions reject_rbl_client,relays.mail-abuse.org
OK
>>> client spike.porcupine.org 168.100.189.2
OK
>>> client foo 127.0.0.2
OK
>>> rcpt rname@rdomain
-./smtpd_check: <queue id>: reject: RCPT from foo[127.0.0.2]: 554 Service unavailable; Client host [127.0.0.2] blocked using socks.relays.osirusoft.com; Proxy List Test; from=<sname@sdomain> to=<rname@rdomain> proto=SMTP helo=<foobar>
-554 Service unavailable; Client host [127.0.0.2] blocked using socks.relays.osirusoft.com; Proxy List Test
+./smtpd_check: <queue id>: reject: RCPT from foo[127.0.0.2]: 554 Service unavailable; Client host [127.0.0.2] blocked using relays.mail-abuse.org; from=<sname@sdomain> to=<rname@rdomain> proto=SMTP helo=<foobar>
+554 Service unavailable; Client host [127.0.0.2] blocked using relays.mail-abuse.org
>>> #
>>> # RHSBL sender domain name
>>> #
break;
}
}
+ VSTRING_TERMINATE(buf);
/*
* Invalid input: continuing text without preceding text. Allowing this
/*
* Done.
*/
- VSTRING_TERMINATE(buf);
return (LEN(buf) > 0 ? buf : 0);
}
projects / thirdparty / postfix.git / commitdiff
