From: drh Date: Tue, 10 Jan 2006 20:01:18 +0000 (+0000) Subject: Improved comments on the server and asynchronous I/O demo programs. (CVS 2909) X-Git-Tag: version-3.6.10~3258 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=fe0f75bb1f4e1937f67e2645afdf029af3872495;p=thirdparty%2Fsqlite.git Improved comments on the server and asynchronous I/O demo programs. (CVS 2909) FossilOrigin-Name: c0f47ccbc915f20d56f393383c21b4026785e6a5 --- diff --git a/manifest b/manifest index 5f2a65aef9..06e726519e 100644 --- a/manifest +++ b/manifest @@ -1,5 +1,5 @@ -C More\spedantic\schanges\sto\scomments\sin\sVDBE.\s\sNo\schanges\sto\scode.\s\sTicket\s#1596.\s(CVS\s2908) -D 2006-01-10T19:45:49 +C Improved\scomments\son\sthe\sserver\sand\sasynchronous\sI/O\sdemo\sprograms.\s(CVS\s2909) +D 2006-01-10T20:01:19 F Makefile.in ab3ffd8d469cef4477257169b82810030a6bb967 F Makefile.linux-gcc aee18d8a05546dcf1888bd4547e442008a49a092 F README 9c4e2d6706bdcc3efdd773ce752a8cdab4f90028 @@ -67,7 +67,7 @@ F src/prepare.c 3283bb65b4b217a092c9cbf65014774e6c3a142d F src/printf.c f47a2f4b5387cd2ebb12e9117a1a5d6bd9a2b812 F src/random.c d40f8d356cecbd351ccfab6eaedd7ec1b54f5261 F src/select.c 579cfdd250c5598de7c867134f7d35a2099b1dcc -F src/server.c 42a2bd02eec5018098a96e08f7a923f4965a2b1d +F src/server.c 519e308651e30102dd3d1f4053ac64c14267e44c F src/shell.c 66b073375efbdee19045e7e0cd38b85f9aff71da F src/sqlite.h.in 821b93f918d126c54d9a91fc928434945655edc3 F src/sqliteInt.h d7584dc5b8e15f1732a195ece9e93049ccde35fa @@ -80,7 +80,7 @@ F src/test4.c a8fd681e139e1c61f22a77d07fc3a99cb28fff3f F src/test5.c 7162f8526affb771c4ed256826eee7bb9eca265f F src/test6.c 74d91b487c68154156eded457925d96aa2a3fdbb F src/test7.c bfe36c62cae189509660acfeeb891ffb9da8ef0c -F src/test_async.c 9733deb7fefa18a3596e5234c1ef05b4685c6ad7 +F src/test_async.c 6776f5027ca6378c116ff5ccc2fe41b908e33772 F src/tokenize.c 196486012c871cdcad6cc84a820cc988603f1b9d F src/trigger.c 883b5f3b97137fbe417e3337c3fa20ac8e9c1ae5 F src/update.c cd8ad5bb1a29f2056347481308fca4a59f2f4764 @@ -340,7 +340,7 @@ F www/tclsqlite.tcl bb0d1357328a42b1993d78573e587c6dcbc964b9 F www/vdbe.tcl 87a31ace769f20d3627a64fa1fade7fed47b90d0 F www/version3.tcl a99cf5f6d8bd4d5537584a2b342f0fb9fa601d8b F www/whentouse.tcl 97e2b5cd296f7d8057e11f44427dea8a4c2db513 -P 511ac9db12ad84bb02d84568b75fc65cef661e88 -R dc7dbd20c2b4f7e0aa62e74e583aeae0 +P 1cf6855430352ffbf921a977186345d7272fe272 +R c77d71bb3bbb59cc1b67cf59dbab5a91 U drh -Z 55bfd4e79be2428d625d90a19139bf0e +Z 85b6b185878e0427b46314ea83850303 diff --git a/manifest.uuid b/manifest.uuid index 890cd71f9a..b1df334272 100644 --- a/manifest.uuid +++ b/manifest.uuid @@ -1 +1 @@ -1cf6855430352ffbf921a977186345d7272fe272 \ No newline at end of file +c0f47ccbc915f20d56f393383c21b4026785e6a5 \ No newline at end of file diff --git a/src/server.c b/src/server.c index 0c9ec81e05..20399eacb9 100644 --- a/src/server.c +++ b/src/server.c @@ -62,8 +62,7 @@ ** ** Note: The extra features of version 3.3.0 described by points (2) ** through (4) above are only available if you compile without the -** option -DSQLITE_OMIT_SHARED_CACHE. For reasons of backwards -** compatibility, SQLite is compile with this option by default. +** option -DSQLITE_OMIT_SHARED_CACHE. ** ** Here is how the client/server approach works: The database server ** thread is started on this procedure: @@ -256,7 +255,7 @@ static void sendToServer(SqlMessage *pMsg){ ** sqlite3_close ** ** Clients should use the following client-side routines instead of -** the core routines. +** the core routines above. ** ** sqlite3_client_open ** sqlite3_client_prepare diff --git a/src/test_async.c b/src/test_async.c index 63fee5ba3a..7ce6c82884 100644 --- a/src/test_async.c +++ b/src/test_async.c @@ -11,33 +11,100 @@ ************************************************************************* ** ** This file contains an example implementation of an asynchronous IO -** backend for SQLite. It is used to test that the concept of asynchronous -** IO in SQLite is valid. +** backend for SQLite. +** +** WHAT IS ASYNCHRONOUS I/O? +** +** With asynchronous I/O, write requests are handled by a separate thread +** running in the background. This means that the thread that initiates +** a database write does not have to wait for (sometimes slow) disk I/O +** to occur. The write seems to happen very quickly, though in reality +** it is happening at its usual slow pace in the background. +** +** Asynchronous I/O appears to give better responsiveness, but at a price. +** You lose the Durable property. With the default I/O backend of SQLite, +** once a write completes, you know that the information you wrote is +** safely on disk. With the asynchronous I/O, this is no the case. If +** your program crashes or if you take a power lose after the database +** write but before the asynchronous write thread has completed, then the +** database change might never make it to disk and the next user of the +** database might not see your change. +** +** You lose Durability with asynchronous I/O, but you still retain the +** other parts of ACID: Atomic, Consistent, and Isolated. Many +** appliations get along fine without the Durablity. +** +** HOW IT WORKS +** +** Asynchronous I/O works by overloading the OS-layer disk I/O routines +** with modified versions that store the data to be written in queue of +** pending write operations. Look at the asyncEnable() subroutine to see +** how overloading works. Six os-layer routines are overloaded: +** +** sqlite3OsOpenReadWrite; +** sqlite3OsOpenReadOnly; +** sqlite3OsOpenExclusive; +** sqlite3OsDelete; +** sqlite3OsFileExists; +** sqlite3OsSyncDirectory; +** +** The original implementations of these routines are saved and are +** used by the writer thread to do the real I/O. The substitute +** implementations typically put the I/O operation on a queue +** to be handled later by the writer thread, though read operations +** must be handled right away, obviously. +** +** Asynchronous I/O is disabled by setting the os-layer interface routines +** back to their original values. +** +** LIMITATIONS +** +** This demonstration code is deliberately kept simple in order to keep +** the main ideas clear and easy to understand. Real applications that +** want to do asynchronous I/O might want to add additional capabilities. +** For example, in this demonstration if writes are happening at a steady +** stream that exceeds the I/O capability of the background writer thread, +** the queue of pending write operations will grow without bound until we +** run out of memory. Users of this technique may want to keep track of +** the quantity of pending writes and stop accepting new write requests +** when the buffer gets to be too big. */ #include "sqliteInt.h" #include "os.h" #include +/* If the THREADSAFE macro is not set, assume that it is turned off. */ #ifndef THREADSAFE # define THREADSAFE 0 #endif /* ** This test uses pthreads and hence only works on unix and with -** a threadsafe build of SQLite. +** a threadsafe build of SQLite. It also requires that the redefinable +** I/O feature of SQLite be turned on. This feature is turned off by +** default. If a required element is missing, almost all of the code +** in this file is commented out. */ #if OS_UNIX && THREADSAFE && defined(SQLITE_ENABLE_REDEF_IO) +/* +** This demo uses pthreads. If you do not have a pthreads implementation +** for your operating system, you will need to recode the threading +** logic. +*/ #include #include +/* Useful macros used in several places */ #define MIN(x,y) ((x)<(y)?(x):(y)) #define MAX(x,y) ((x)>(y)?(x):(y)) +/* Forward references */ typedef struct AsyncWrite AsyncWrite; typedef struct AsyncFile AsyncFile; +/* Enable for debugging */ #if 0 # define TRACE(X,Y) \ fprintf(stderr,"THRD=%d: ", (int)pthread_self()); \ @@ -46,11 +113,6 @@ typedef struct AsyncFile AsyncFile; # define TRACE(X,Y) /* noop */ #endif -/* -** TODO: -** * File locks... -*/ - /* ** THREAD SAFETY NOTES ** @@ -62,7 +124,7 @@ typedef struct AsyncFile AsyncFile; ** * The file handles from the underlying system are assumed not to ** be thread safe. ** -** * See the last paragraph under "sqlite3_async_flush() Threads" for +** * See the last two paragraphs under "The Writer Thread" for ** an assumption to do with file-handle synchronization by the Os. ** ** File system operations (invoked by SQLite thread): @@ -74,39 +136,47 @@ typedef struct AsyncFile AsyncFile; ** ** File handle operations (invoked by SQLite thread): ** -** The following operations add an entry to the global write-op list. They -** prepare the entry, acquire the mutex momentarily while list pointers are -** manipulated to insert the new entry, and release the mutex. -** ** asyncWrite, asyncClose, asyncTruncate, asyncSync, ** asyncSetFullSync, asyncOpenDirectory. ** -** Read operations. Both of these read from both the underlying file and -** the write-op list. So we grab the mutex for the whole call (even -** while performing a blocking read on the file). +** The operations above add an entry to the global write-op list. They +** prepare the entry, acquire the async.queueMutex momentarily while +** list pointers are manipulated to insert the new entry, then release +** the mutex and signal the writer thread to wake up in case it happens +** to be asleep. +** ** ** asyncRead, asyncFileSize. +** +** Read operations. Both of these read from both the underlying file +** first then adjust their result based on pending writes in the +** write-op queue. So async.queueMutex is held for the duration +** of these operations to prevent other threads from changing the +** queue in mid operation. ** -** These locking primitives become no-ops. Files are always opened for -** exclusive access when using this IO backend: -** +** ** asyncLock, asyncUnlock, asyncLockState, asyncCheckReservedLock ** +** These locking primitives become no-ops. Files are always opened for +** exclusive access when using this IO backend. +** +** +** asyncFileHandle. +** ** The sqlite3OsFileHandle() function is currently only used when ** debugging the pager module. Unless sqlite3OsClose() is called on the ** file (shouldn't be possible for other reasons), the underlying ** implementations are safe to call without grabbing any mutex. So we just -** go ahead and call it no matter what any other thread is doing. +** go ahead and call it no matter what any other threads are doing. ** -** asyncFileHandle. +** +** asyncSeek. ** ** Calling this method just manipulates the AsyncFile.iOffset variable. ** Since this variable is never accessed by writer thread, this ** function does not require the mutex. Actual calls to OsSeek() take ** place just before OsWrite() or OsRead(), which are always protected by ** the mutex. -** -** asyncSeek. ** ** The writer thread: ** @@ -123,7 +193,7 @@ typedef struct AsyncFile AsyncFile; ** The async.queueMutex is always held during the test, and when the entry is removed from the head ** of the write-op list. Sometimes it is held for the interim -** period (while the IO is performed), and sometimes it is +** period (while the IO is performed), and sometimes it is ** relinquished. It is relinquished if (a) the IO op is an ** ASYNC_CLOSE or (b) when the file handle was opened, two of ** the underlying systems handles were opened on the same @@ -186,6 +256,9 @@ static struct TestAsyncStaticData { #define ASYNC_SYNCDIRECTORY 9 /* +** Entries on the write-op queue are instances of the AsyncWrite +** structure, defined here. +** ** The interpretation of the iOffset and nByte variables varies depending ** on the value of AsyncWrite.op: ** @@ -248,15 +321,19 @@ struct AsyncFile { /* ** Add an entry to the end of the global write-op list. pWrite should point -** to an AsyncWrite structure allocated using sqliteMalloc(). A future call -** to sqlite3_async_flush() is responsible for calling sqliteFree(). +** to an AsyncWrite structure allocated using sqlite3OsMalloc(). The writer +** thread will call sqlite3OsFree() to free the structure after the specified +** operation has been completed. ** -** Once an AsyncWrite structure has been added to the list, it must not be -** read or modified by the caller (in case another thread calls -** sqlite3_async_flush() ). +** Once an AsyncWrite structure has been added to the list, it becomes the +** property of the writer thread and must not be read or modified by the +** caller. */ static void addAsyncWrite(AsyncWrite *pWrite){ + /* We must hold the queue mutex in order to modify the queue pointers */ pthread_mutex_lock(&async.queueMutex); + + /* Add the record to the end of the write-op queue */ assert( !pWrite->pNext ); if( async.pQueueLast ){ assert( async.pQueueFirst ); @@ -266,7 +343,12 @@ static void addAsyncWrite(AsyncWrite *pWrite){ } async.pQueueLast = pWrite; TRACE("PUSH %p\n", pWrite); + + /* Drop the queue mutex */ pthread_mutex_unlock(&async.queueMutex); + + /* The writer thread might have been idle because there was nothing + ** on the write-op queue for it to do. So wake it up. */ pthread_cond_signal(&async.queueSignal); } @@ -358,8 +440,7 @@ static void asyncSetFullSync(OsFile *id, int value){ /* ** Read data from the file. First we read from the filesystem, then adjust ** the contents of the buffer based on ASYNC_WRITE operations in the -** write-op queue. Todo: Do we need to think about ASYNC_TRUNCATE in -** this method as well? +** write-op queue. ** ** This method holds the mutex from start to finish. */ @@ -478,6 +559,10 @@ static int asyncFileHandle(OsFile *id){ return sqlite3OsFileHandle(((AsyncFile *)id)->pBaseRead); } +/* +** No file locking occurs with this version of the asynchronous backend. +** So the locking routines are no-ops. +*/ static int asyncLock(OsFile *id, int lockType){ return SQLITE_OK; } @@ -502,8 +587,8 @@ static int asyncLockState(OsFile *id){ /* ** The following variables hold pointers to the original versions of -** certain OS-layer interface routines - routines that this module -** overrides. +** OS-layer interface routines that are overloaded in order to create +** the asynchronous I/O backend. */ static int (*xOrigOpenReadWrite)(const char*, OsFile**, int*) = 0; static int (*xOrigOpenExclusive)(const char*, OsFile**, int) = 0; @@ -512,12 +597,15 @@ static int (*xOrigDelete)(const char*) = 0; static int (*xOrigFileExists)(const char*) = 0; static int (*xOrigSyncDirectory)(const char*) = 0; - +/* +** This routine does most of the work of opening a file and building +** the OsFile structure. +*/ static int asyncOpenFile( - const char *zName, - OsFile **pFile, - OsFile *pBaseRead, - int openSecondFile + const char *zName, /* The name of the file to be opened */ + OsFile **pFile, /* Put the OsFile structure here */ + OsFile *pBaseRead, /* The real OsFile from the real I/O routine */ + int openForWriting /* Open a second file handle for writing if true */ ){ int rc; AsyncFile *p; @@ -540,7 +628,7 @@ static int asyncOpenFile( asyncCheckReservedLock }; - if( openSecondFile && SQLITE_ASYNC_TWO_FILEHANDLES ){ + if( openForWriting && SQLITE_ASYNC_TWO_FILEHANDLES ){ int dummy; rc = xOrigOpenReadWrite(zName, &pBaseWrite, &dummy); if( rc!=SQLITE_OK ){