GGLinnk/bees
1
0
Fork 0
mirror of https://github.com/Zygo/bees.git synced 2025-05-17 21:35:45 +02:00
Code Packages

bees: soft-limit computed thread counts to 8

Browse Source 
https://github.com/Zygo/bees/issues/91 describes problems encountered
when running bees on systems with many CPU cores.

Limit the computed number of threads (using --thread-factor or the
default) to a maximum of 8 (i.e. the number of logical cores in a modern
laptop).  Users can override the limit by using --thread-count.

Signed-off-by: Zygo Blaxell <bees@furryterror.org>
This commit is contained in:
Zygo Blaxell 2018-11-21 21:28:38 -05:00
parent d9c788d30a
commit 34b04f4255
4 changed files with 21 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
docs/btrfs-kernel.md
View File

@ -62,6 +62,12 @@ Minor kernel problems with workarounds:
Note `btrfs receive` is not affected. It is OK to run bees with no
workarounds on a filesystem that receives btrfs snapshots.
* **Systems with many CPU cores** may [lock up when bees runs with one
worker thread for every core](https://github.com/Zygo/bees/issues/91).
bees limits the number of threads it will try to create based on
detected CPU core count. Users may override this limit with the
[`--thread-count` option](options.md).
Older kernels:
* Older kernels have various data corruption and deadlock/hang issues

8
docs/options.md
View File

@ -5,7 +5,7 @@
* `--thread-count COUNT` or `-c`
Specify maximum number of worker threads. Overrides `--thread-factor`
(`-C`) and default/autodetected values.
(`-C`), default/autodetected values, and the hardcoded thread limit.
* `--thread-factor FACTOR` or `-C`
@ -16,6 +16,12 @@
below 1.0 to leave some cores idle, or above 1.0 if there are more
disks than CPUs in the filesystem.
If the computed thread count is higher than `BEES_DEFAULT_THREAD_LIMIT`
(currently 8), then only that number of threads will be created.
This limit can be overridden by the `--thread-count` option; however,
be aware that there are [kernel issues with systems that have many CPU
cores](btrfs-kernel.md) when users try to run bees on all of them.
* `--loadavg-target LOADAVG` or `-g`
Specify load average target for dynamic worker threads. Default is

5
src/bees.cc
View File

@ -786,6 +786,11 @@ bees_main(int argc, char *argv[])
thread_factor = BEES_DEFAULT_THREAD_FACTOR;
}
thread_count = max(1U, static_cast<unsigned>(ceil(thread::hardware_concurrency() * thread_factor)));
if (thread_count > BEES_DEFAULT_THREAD_LIMIT) {
BEESLOGNOTICE("Limiting computed thread count to " << BEES_DEFAULT_THREAD_LIMIT);
BEESLOGNOTICE("Use --thread-count to override this limit");
thread_count = BEES_DEFAULT_THREAD_LIMIT;
}
}
if (load_target != 0) {

3
src/bees.h
View File

@ -85,6 +85,9 @@ const size_t BEES_OPEN_FILE_LIMIT = (BEES_FILE_FD_CACHE_SIZE + BEES_ROOT_FD_CACH
// Worker thread factor (multiplied by detected number of CPU cores)
const double BEES_DEFAULT_THREAD_FACTOR = 1.0;
// Don't use more than this number of threads unless explicitly configured
const size_t BEES_DEFAULT_THREAD_LIMIT = 8;
// Log warnings when an operation takes too long
const double BEES_TOO_LONG = 5.0;