Add file doc/vdbesort-memory.md, containing notes on the way vdbesort.c uses heap...

FossilOrigin-Name: 07f53899a929cce93f17c0332819610c9517c7e54e00092af20efd746cf84438

diff --git a/doc/vdbesort-memory.md b/doc/vdbesort-memory.md
new file mode 100644 (file)
index 0000000..5c3dd62
--- /dev/null
+++ b/doc/vdbesort-memory.md
@@ -0,0 +1,49 @@
+
+20-11-2020
+
+# Memory Allocation In vdbesort.c
+
+Memory allocation is slightly different depending on:
+
+  * whether or not SQLITE_CONFIG_SMALL_MALLOC is set, and
+  * whether or not worker threads are enabled.
+
+## SQLITE_CONFIG_SMALL_MALLOC=0
+
+Assuming SQLITE_CONFIG_SMALL_MALLOC is not set, keys passed to the sorter are
+added to an in-memory buffer. This buffer is grown using sqlite3Realloc() as
+required it reaches the size configured for the main pager cache using "PRAGMA
+cache_size". i.e. if the user has executed "PRAGMA main.cache_size = -2048",
+then this buffer is allowed to grow up to 2MB in size.
+
+Once the buffer has grown to its threshold, keys are sorted and written to
+a temp file. If worker threads are not enabled, this is the only significant
+allocation the sorter module makes. After keys are sorted and flushed out to
+the temp file, the buffer is reused to accumulate the next batch of keys.
+
+If worker threads are available, then the buffer is passed to a worker thread
+to sort and flush once it is full, and a new buffer allocated to allow the
+main thread to continue to accumulate keys. Buffers are reused once they
+have been flushed, so in this case at most (nWorker+1) buffers are allocated
+and used, where nWorker is the number of configured worker threads.
+
+There are no other significant users of heap memory in the sorter module. 
+Once sorted buffers of keys have been flushed to disk, they are read back
+either by mapping the file (via sqlite3_file.xFetch()) or else read back
+in one page at a time.
+
+All buffers are allocated by the main thread. A sorter object is associated
+with a single database connection, to which it holds a pointer.
+
+## SQLITE_CONFIG_SMALL_MALLOC=1
+
+This case is similar to the above, except that instead of accumulating
+multiple keys in a single large buffer, sqlite3VdbeSorterWrite() stores
+keys in a regular heap-memory linked list (one allocation per element).
+List elements are freed as they are flushed to disk, either by the main
+thread or by a worker thread.
+
+Each time a key is added the sorter (and an allocation made),
+sqlite3HeapNearlyFull() is called. If it returns true, the current
+list of keys is flushed to a temporary file, even if it has not yet
+reached the size threshold.

diff --git a/manifest b/manifest
index 4fab75a14489c478246c3a25d445f0ad6d739c2d..4fd4950cf5f39ee5297c3ed3e029f15a78e0e2e3 100644 (file)
--- a/manifest
+++ b/manifest
@@ -1,5 +1,5 @@
-C The\sxFullPathname\smethod\sof\sthe\sunix\sVFS\snow\snormalizes\sthe\sfilenames\sby\nomitting\ssurplus\s"/",\s"/./"\sand\s"/../"\sstrings.
-D 2020-11-19T21:12:08.560
+C Add\sfile\sdoc/vdbesort-memory.md,\scontaining\snotes\son\sthe\sway\svdbesort.c\suses\sheap\smemory.
+D 2020-11-20T14:16:41.814
 F .fossil-settings/empty-dirs dbb81e8fc0401ac46a1491ab34a7f2c7c0452f2f06b54ebb845d024ca8283ef1
 F .fossil-settings/ignore-glob 35175cdfcf539b2318cb04a9901442804be81cd677d8b889fcc9149c21f239ea
 F LICENSE.md df5091916dbb40e6e9686186587125e1b2ff51f022cc334e886c19a0e9982724
@@ -41,6 +41,7 @@ F doc/F2FS.txt c1d4a0ae9711cfe0e1d8b019d154f1c29e0d3abfe820787ba1e9ed7691160fcd
 F doc/lemon.html c5d8ba85ac1daef7be8c2d389899480eb62451ff5c09b0c28ff8157bb8770746
 F doc/pager-invariants.txt 27fed9a70ddad2088750c4a2b493b63853da2710
 F doc/trusted-schema.md 33625008620e879c7bcfbbfa079587612c434fa094d338b08242288d358c3e8a
+F doc/vdbesort-memory.md 4da2639c14cd24a31e0af694b1a8dd37eaf277aff3867e9a8cc14046bc49df56
 F doc/vfs-shm.txt e101f27ea02a8387ce46a05be2b1a902a021d37a
 F doc/wal-lock.md 781726aaba20bafeceb7ba9f91d5c98c6731691b30c954e37cf0b49a053d461d
 F ext/README.md fd5f78013b0a2bc6f0067afb19e6ad040e89a10179b4f6f03eee58fac5f169bd
@@ -1885,8 +1886,7 @@ F vsixtest/vsixtest.tcl 6a9a6ab600c25a91a7acc6293828957a386a8a93
 F vsixtest/vsixtest.vcxproj.data 2ed517e100c66dc455b492e1a33350c1b20fbcdc
 F vsixtest/vsixtest.vcxproj.filters 37e51ffedcdb064aad6ff33b6148725226cd608e
 F vsixtest/vsixtest_TemporaryKey.pfx e5b1b036facdb453873e7084e1cae9102ccc67a0
-P 842c432772e6cd8464cdb7bfdb38789adeea9aa9e0486d4034cc9841f085f517 6f0d5f0949130a4688319adf05d37c5ccd683ca9ad2dacb7ce433a5fe18ac3c3
-R 9bca7f34bcf3572651c390350ebafdef
-T +closed 6f0d5f0949130a4688319adf05d37c5ccd683ca9ad2dacb7ce433a5fe18ac3c3
-U drh
-Z 8c74c867e3ffdcd79c4ff4d10ad118b3
+P c38dec6f52c01614c1bee8356daf0fcd9f708d029116e9bff51e06719a730dde
+R b9b1a27eff3b385f6d301a9808892950
+U dan
+Z d17ac9b00e741eb4707d9deb9c3699e7

diff --git a/manifest.uuid b/manifest.uuid
index c187ecc61425aba5b1443fc8878901685ef55235..d4efb20104b85f5d7ba7f279a1feb21a3d100ad0 100644 (file)
--- a/manifest.uuid
+++ b/manifest.uuid
@@ -1 +1 @@
-c38dec6f52c01614c1bee8356daf0fcd9f708d029116e9bff51e06719a730dde
\ No newline at end of file
+07f53899a929cce93f17c0332819610c9517c7e54e00092af20efd746cf84438
\ No newline at end of file