bees: soft-limit computed thread counts to 8

https://github.com/Zygo/bees/issues/91 describes problems encountered
when running bees on systems with many CPU cores.

Limit the computed number of threads (using --thread-factor or the
default) to a maximum of 8 (i.e. the number of logical cores in a modern
laptop).  Users can override the limit by using --thread-count.

Signed-off-by: Zygo Blaxell <bees@furryterror.org>

docs/btrfs-kernel.md

@@ -62,6 +62,12 @@ Minor kernel problems with workarounds:
   Note `btrfs receive` is not affected.  It is OK to run bees with no
   workarounds on a filesystem that receives btrfs snapshots.
 
+* **Systems with many CPU cores** may [lock up when bees runs with one
+  worker thread for every core](https://github.com/Zygo/bees/issues/91).
+  bees limits the number of threads it will try to create based on
+  detected CPU core count.  Users may override this limit with the
+  [`--thread-count` option](options.md).
+
 Older kernels:
 
 * Older kernels have various data corruption and deadlock/hang issues

docs/options.md

@@ -5,7 +5,7 @@
 * `--thread-count COUNT` or `-c`
 
  Specify maximum number of worker threads.  Overrides `--thread-factor`
- (`-C`) and default/autodetected values.
+ (`-C`), default/autodetected values, and the hardcoded thread limit.
 
 * `--thread-factor FACTOR` or `-C`
 
@@ -16,6 +16,12 @@
  below 1.0 to leave some cores idle, or above 1.0 if there are more
  disks than CPUs in the filesystem.
 
+ If the computed thread count is higher than `BEES_DEFAULT_THREAD_LIMIT`
+ (currently 8), then only that number of threads will be created.
+ This limit can be overridden by the `--thread-count` option; however,
+ be aware that there are [kernel issues with systems that have many CPU
+ cores](btrfs-kernel.md) when users try to run bees on all of them.
+
 * `--loadavg-target LOADAVG` or `-g`
 
  Specify load average target for dynamic worker threads.  Default is

src/bees.cc

@@ -786,6 +786,11 @@ bees_main(int argc, char *argv[])
 			thread_factor = BEES_DEFAULT_THREAD_FACTOR;
 		}
 		thread_count = max(1U, static_cast<unsigned>(ceil(thread::hardware_concurrency() * thread_factor)));
+		if (thread_count > BEES_DEFAULT_THREAD_LIMIT) {
+			BEESLOGNOTICE("Limiting computed thread count to " << BEES_DEFAULT_THREAD_LIMIT);
+			BEESLOGNOTICE("Use --thread-count to override this limit");
+			thread_count = BEES_DEFAULT_THREAD_LIMIT;
+		}
 	}
 
 	if (load_target != 0) {

src/bees.h

@@ -85,6 +85,9 @@ const size_t BEES_OPEN_FILE_LIMIT = (BEES_FILE_FD_CACHE_SIZE + BEES_ROOT_FD_CACH
 // Worker thread factor (multiplied by detected number of CPU cores)
 const double BEES_DEFAULT_THREAD_FACTOR = 1.0;
 
+// Don't use more than this number of threads unless explicitly configured
+const size_t BEES_DEFAULT_THREAD_LIMIT = 8;
+
 // Log warnings when an operation takes too long
 const double BEES_TOO_LONG = 5.0;