GGLinnk/bees
1
0
Fork 0
mirror of https://github.com/Zygo/bees.git synced 2025-05-17 21:35:45 +02:00
Code Packages

bees: soft-limit computed thread counts to 8

Browse Source 
https://github.com/Zygo/bees/issues/91 describes problems encountered
when running bees on systems with many CPU cores.

Limit the computed number of threads (using --thread-factor or the
default) to a maximum of 8 (i.e. the number of logical cores in a modern
laptop).  Users can override the limit by using --thread-count.

Signed-off-by: Zygo Blaxell <bees@furryterror.org>
This commit is contained in:
Zygo Blaxell 2018-11-21 21:28:38 -05:00 committed by Kai Krakow
parent df640062e7
commit 04dbfd5bf1
No known key found for this signature in database
GPG Key ID: 046FAC3028D76321
3 changed files with 21 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
README.md
View File

@ -336,6 +336,12 @@ Unfixed kernel bugs (as of 4.14.34) with workarounds in Bees:
or prealloc. Bees avoids feedback loops this can generate while
attempting to replace extents over 16MB in length.
* **Systems with many CPU cores** may [lock up when bees runs with one
worker thread for every core](https://github.com/Zygo/bees/issues/91).
bees limits the number of threads it will try to create based on
detected CPU core count. Users may override this limit with the
[`--thread-count` option](options.md).
Not really bugs, but gotchas nonetheless:
* If a process holds a directory FD open, the subvol containing the
@ -536,12 +542,18 @@ Command Line Options
* --thread-count (-c) COUNT
* Specify maximum number of worker threads for scanning. Overrides
--thread-factor (-C) and default/autodetected values.
--thread-factor (-C) and default/autodetected values,
and the hardcoded thread limit.
* --thread-factor (-C) FACTOR
* Specify ratio of worker threads to CPU cores. Overridden by --thread-count (-c).
Default is 1.0, i.e. 1 worker thread per detected CPU. Use values
below 1.0 to leave some cores idle, or above 1.0 if there are more
disks than CPUs in the filesystem.
If the computed thread count is higher than `BEES_DEFAULT_THREAD_LIMIT`
(currently 8), then only that number of threads will be created.
This limit can be overridden by the `--thread-count` option; however,
be aware that there are kernel issues with systems that have many CPU
cores when users try to run bees on all of them.
* --loadavg-target (-g) LOADAVG
* Specify load average target for dynamic worker threads.
Threads will be started or stopped subject to the upper limit imposed

5
src/bees.cc
View File

@ -785,6 +785,11 @@ bees_main(int argc, char *argv[])
thread_factor = BEES_DEFAULT_THREAD_FACTOR;
}
thread_count = max(1U, static_cast<unsigned>(ceil(thread::hardware_concurrency() * thread_factor)));
if (thread_count > BEES_DEFAULT_THREAD_LIMIT) {
BEESLOGNOTICE("Limiting computed thread count to " << BEES_DEFAULT_THREAD_LIMIT);
BEESLOGNOTICE("Use --thread-count to override this limit");
thread_count = BEES_DEFAULT_THREAD_LIMIT;
}
}
if (load_target != 0) {

3
src/bees.h
View File

@ -85,6 +85,9 @@ const size_t BEES_OPEN_FILE_LIMIT = (BEES_FILE_FD_CACHE_SIZE + BEES_ROOT_FD_CACH
// Worker thread factor (multiplied by detected number of CPU cores)
const double BEES_DEFAULT_THREAD_FACTOR = 1.0;
// Don't use more than this number of threads unless explicitly configured
const size_t BEES_DEFAULT_THREAD_LIMIT = 8;
// Log warnings when an operation takes too long
const double BEES_TOO_LONG = 5.0;