Adding initial notes on AsyncCalls and Comm APIs.

More work is needed to integrate source code and the notes.

diff --git a/src/AsyncCalls.dox b/src/AsyncCalls.dox
new file mode 100644 (file)
index 0000000..b0be201
--- /dev/null
+++ b/src/AsyncCalls.dox
@@ -0,0 +1,91 @@
+/**
+\defgroup AsyncCalls Asynchronous Calls
+\ingroup Components
+ 
+
+\section Terminology Terminology
+
+- \b Call: an AsyncCall object meant to be delivered from the Caller to the
+  Recipient.
+- \b Recipient: the code being the destination of the Call. The specific
+  function receiving the call is often called "handler".
+- \b Caller: the code scheduling or placing the Call.
+- \b Creator: the code creating the Call.
+- \b Dialing: the final act of delivering the Call to the Recipient.
+- \b Canceling: preventing the call from being dialed in the future.
+- <b>Pending call</b>: the call which has been scheduled but no attempt to dial
+  it has been made.
+- <b>Live call</b>: the in-progress call that has been dialed and the Recipient
+  (receiving) function has not finished yet.
+
+
+\section Life Typical life cycle
+
+-# Creator creates a Call object and stores its pointer, possibly in a
+temporary variable. If Creator is the future Recipient of the Call, it
+is customary to talk about a "callback". In such context, Creator gives
+the Call pointer to some other code (the future Caller). Calls are
+refcounted and the AsyncCall::Pointer class must be used to point and
+share them.
+-# Time passes and something triggers the need to dial the call.
+-# Caller schedules (a.k.a, places) the Call. The Call is placed in the
+AsyncCallQueue. The queue preserves the scheduling call order.
+-# Time passes and it is now time to dial the call.
+-# AsyncCallQueue checks whether the call can be dialed. If yes,
+AsyncCallQueue dials the Call and the Recipient receives it. Otherwise,
+only a debugging message is logged (if enabled). Dialing a call involves
+several state checks at several stages, depending on the Call type.
+-# The Call is alive while the Recipient is handling it. That state ends
+when the Recipient code (a.k.a., handler) returns. At that point, the
+AsyncCallQueue proceeds to handle other calls.
+-# The Call object is destroyed when the last reference to it is gone.
+One should not try to invalidate or delete the call.
+
+
+\section Rules Basic rules
+
+- Callbacks must be called: If a Caller has a Call pointer as a
+callback, the Caller must make that Call when conditions are right
+(possibly with a call-specific error indicator). Whether that Call will
+be dialed is irrelevant here.
+
+- Unwanted calls must be canceled: If Recipient may not be able to
+handle the call back, then it must cancel it.  Whether that Call will be
+scheduled is irrelevant here. If the Recipient has an AsyncCall pointer,
+calling AsyncCall::cancel is sufficient, but the code should use
+call-specific APIs when possible (e.g., comm_read_cancel or comm_close).
+
+- Processed calls should be forgotten: If you scheduled, received, or
+cancel the call, set its pointer to NULL. The Caller should forget the
+call after scheduling it. the recipient should forget the call when it
+receives or cancels it. These are just cleanup rules to avoid
+double-scheduling or double-cancellation as well as situations where the
+code assumes it can still change a pending Call (which will not work for
+SMP code).
+
+- The calls are received in the order they were scheduled.
+
+- The Recipient is guaranteed to receive at most one Call at a time. An
+attempt to dial call B while some call A is alive is a bug, and call B
+will not be dialed.
+
+
+\section Canceling Call Canceling
+
+- A call may be canceled at any time.
+
+- A canceled call will not be dialed. 
+
+- Cancellation has immediate effect: The call will never be dialed
+after cancel() is called
+
+- Cancellation by non-Recipients is discouraged because it may not work
+in an SMP environment.
+
+- A late call cancellation is cancellation that is performed after the Call
+has already been dialed (i.e., the call is or was alive at the time of the
+cancellation).  Late cancellation is a no-op. It is not a bug to cancel
+something late, but the canceling code should be aware of that possibility.
+
+
+*/

diff --git a/src/Comm.dox b/src/Comm.dox
new file mode 100644 (file)
index 0000000..55eb1ec
--- /dev/null
+++ b/src/Comm.dox
@@ -0,0 +1,182 @@
+/**
+\defgroup Comm Comm Module
+\ingroup Components
+ 
+
+\section Basic Basic Comm API principles
+
+  \par
+  Comm API supports four major kinds of socket-related operations:
+  accept, connect, read, and write. Sockets are identified by their
+  descriptors.
+
+  \par
+  A Comm user requests its interest in operations by calling an
+  appropriate API function (e.g., comm_read()) and optionally providing
+  a notification "callback" (a.k.a., handler). For a given descriptor,
+  there can be no more than one registered user per operation.
+
+  \par
+  In addition to the four operations listed above, a user can register
+  to be notified when a given descriptor is closed.
+
+  \par
+  When a Comm operation completes, all users that registered the
+  corresponding handler are notified. When a descriptor is closed, all
+  users that registered any callback for the descriptor are notified
+  (this will change though, see "Anticipated changes" below).
+
+  \par
+  All Comm notifications are asynchronous, performed using the
+  AsyncCall API.
+
+  \par
+  Notifications for four operations listed above are scheduled in the
+  order of the corresponding operations completion. User code can assume
+  that the operation has completed (possibly with an error) only after
+  receiving a notification. Until then, I/O resources such as buffers
+  must remain available for the operation.
+
+  \par
+  Notifications related to closing of a descriptor are documented
+  separately.
+
+
+\section IO-cancel I/O cancellation
+
+  \par
+  To cancel an interest in a read operation, call comm_read_cancel()
+  with an AsyncCall object. This call guarantees that the passed Call
+  will be canceled (see the AsyncCall API for call cancellation
+  definitions and details). Naturally, the code has to store the
+  original read callback Call pointer to use this interface. 
+
+  \par
+  The comm_read_cancel() call does not guarantee that the read operation
+  has not already happen.
+
+  \par
+  The comm_read_cancel() call guarantees that the read operation will not
+  start for read operations that are performed by Comm (i.e., where read
+  start is controlled by Comm).  There is no such guarantee for
+  asynchronous read operations scheduled by Comm but started by the
+  operating system or other threads.
+
+  \par
+  The above applies to comm_read_cancel() interface with an AsyncCall
+  object as a parameter. You cannot reliably cancel an interest in read
+  operation using the old comm_read_cancel() call that uses a function
+  pointer. The handler may get even called after old comm_read_cancel()
+  was called.
+
+  \par
+  It is OK to call comm_read_cancel (both flavors) at any time as long
+  as the descriptor has not been closed and there is either no read
+  interest registered or the passed parameters match the registered
+  ones. If the descriptor has been closed, the behavior is undefined.
+  Otherwise, if parameters do not match, you get an assertion.
+
+  \par
+  To cancel Comm operations other than read, close the descriptor with
+  comm_close().
+
+
+\section comm-close Descriptor closing with comm_close
+
+  \par
+  The comm_close() function does not close the descriptor but initiates
+  the following closing sequence:
+
+  \par
+      -# The descriptor is placed in a "closing" state.
+      -# The registered read, write, and accept callbacks (if any) are
+         scheduled (in an unspecified order).
+      -# The close callbacks are scheduled (in an unspecified order).
+      -# A call to the internal descriptor closing handler is
+         scheduled.
+
+  \par
+  The "unspecified" order above means that the user should not rely on
+  any specific or deterministic order because the currently hard-coded
+  order may change.
+
+  \par
+  The read, write, and accept notifications (scheduled in step #2
+  above) carry the COMM_ERR_CLOSING error flag. When handling
+  COMM_ERR_CLOSING event, the user code should limit
+  descriptor-related processing, especially Comm calls, because
+  supported Comm functionality is very limited when the descriptor is
+  closing. New code should use the close handlers instead (scheduled
+  in step #3).
+
+  \par
+  The internal closing handler (scheduled in step #4 above) closes the
+  descriptor. When the descriptor is closed, all operations on the
+  descriptor are prohibited and may cause bugs and asserts. Currently,
+  the same descriptor will eventually be reused for another socket and
+  Comm may not notice that a buggy code is still using a stale
+  descriptor, but that may change.
+
+  \par
+  Since all notifications are asynchronous, it is possible for a read
+  or write notification that was scheduled before comm_close() was
+  called to arrive at its destination after comm_close() was called.
+  Such notification will arrive with COMM_ERR_CLOSING flag even though
+  that flag was not set at the time of the I/O (and the I/O may have
+  been successful). This behavior may change.
+
+
+\section Future Anticipated changes and preparation recommendations
+
+  \par
+  This section lists anticipated Comm API changes and provides
+  recommendations for developers writing new (or rewriting old) Comm
+  user code. The changes are listed in a rough order from more likely
+  to less certain and from near-feature to long-term.
+
+  \par
+  The old comm_read_cancel() function that uses a function pointer will be
+  removed as unreliable. Use the AsyncCall-based comm_read_cancel()
+  instead.
+
+  \par
+  COMM_ERR_CLOSING interface will be removed. The read, write, and
+  accept notifications will not be scheduled after comm_close() is
+  called.  New user code should register close handlers instead.
+
+  \par
+  When COMM_ERR_CLOSING interface is removed, pending notifications
+  (if any) will be canceled after comm_close() is called. However, the
+  cancellation may be removed later if Comm is modified to provide safe
+  access to closing descriptors and their fragile state. New user code
+  should continue to assume that it is safe to access Comm in a read,
+  write, and accept handlers.
+
+  \par
+  The old comm_read_cancel() call that uses a function pointer will be
+  removed as unreliable. New user code should use comm_read_cancel() with
+  an AsyncCall parameter.
+
+  \par
+  Comm users may be required to become children of Comm-provided
+  classes, to eliminate the need for a complicated (albeit hidden)
+  hierarchy of Comm callback dialers (see CommCalls.h) and to provide
+  default implementation for common I/O handling cases.  New user code
+  should use methods of AsyncJob-derived classes to handle Comm
+  notifications instead of using stand-alone functions. Additionally, it
+  should not call comm_close() for descriptors it does not "own".
+
+  \par
+  The comm_close() API will be used exclusively for "stop future I/O,
+  schedule a close callback call, and cancel all other callbacks"
+  purposes. New user code should not use comm_close() for the purpose of
+  immediately ending a job.
+
+  \par
+  Raw socket descriptors will be replaced with unique IDs or small
+  objects that help detect stale descriptor/socket usage bugs and
+  encapsulate access to socket-specific information. New user code
+  should treat descriptor integers as opaque objects.
+
+
+ */