Use uint8_t when we mean uint8_t, i.e. vector<uint8_t> instead of
vector<char>.
Add a template parameter instead of vector so we can swap in a
non-copying data type.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
Define a local copy of the header that has fields for the csum type
and length, so we can build in places that haven't caught up to kernel
5.5 headers yet.
The reason why the csum type and length are not unconditionally filled
in eludes me. csum_length is necessarily non-zero, and the cost of
the conditional is worse than the cost of the copy, so the whole flags
dance is a WTF...but it's part of the kernel API now, so it's too late
to NAK it.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
Rewrite Fd using a much simpler named resource template class with
a more straightforward derivation strategy.
Behavior change: we no longer throw an exception while calling get_fd()
on a closed Fd. This does not seem to bother any current callers except
for the tests.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
Use a single static variable located in the library, instead of
having a separate one for each compilation unit.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
If we create an identical .version.cc then don't bother keeping it.
This prevents libcrucible from rebuilding if there are no other changes,
which in turn prevents all the binaries from rebuilding unconditionally.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
Perf blames this operator for >1% of instructions with -O2, and
70% of instructions without -O2.
Let the compiler inline the function.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
The requested size may not match the final size of the container,
so consistently use the container's size after prepare(), not the
requested size.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
Silence the unused variable warning. The compiler is correct, but we
may implement line-level debug at some point in the future, so we
want to keep the member and parameters.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
Get rid of unused template instantiation.
Drop the unused realtime signals from the ntoa table. If in the future
we really need to solve clang's issue with them, we'll address it then.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
uncaught_exception() had only the one valid use case, and it can be
reimplemented by literally calling current_exception() instead.
current_exception() has several valid use cases, so it is not likely
to be deprecated any time soon.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
In version 2.30 glibc added it's own gettid() function. This resulted in
"error: call of overloaded ‘gettid()’ is ambiguous" because gettid()
now exists in both namespace crucible and std.
For now, use explicit references to namespace crucible. This continues
to work with new and old libc without having to test specific library
versions.
At some point, glibc gettid() will be deployed widely enough that we can
remove the crucible version entirely.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
CityHash64 appears to be the fastest available block hashing algorithm
that is good enough for dedupe. It takes much less CPU than the CRC64
function, and avoids hash-collision problems with file formats that use
CRC64 as an integrity check on 4K block boundaries.
Extracted from git://github.com/google/cityhash with the "CRC" hash
functions (which require Intel/AMD CPU support) removed. We don't
need those, and they introduce a new (if only theoretical) build-time
dependency.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
We got away with this because GCC 4.8 (and apparently every GCC prior
to 9) didn't notice or care, and because there is nothing referenced
inside the lambda function body that isn't accessible from any other
kind of function body (i.e. the capture wasn't needed at all).
GCC 9 now enforces what the C++ standard said all along: there is
no need to allow capture-default in this case, so it is not.
Fix by removing the offending capture-default.
Fixes: https://github.com/Zygo/bees/issues/112
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
It is not possible to emulate extent-same by clone in a safe way.
EXTENT_SAME has been supported in btrfs since kernel 3.13, which
is much too old to contemplate running bees on.
Remove this dangerous and unused function.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
We are getting a lot of exceptions when an inline extent is too large for the
TREE_SEARCH_V2 buffer. This disrupts ExtentWalker's extent boundary
search when there is an inline extent at the beginning of a file:
# fiemap foo
Log 0x0..0x1000 Phy 0x0..0x1000 Flags FIEMAP_EXTENT_NOT_ALIGNED|FIEMAP_EXTENT_DATA_INLINE
Log 0x1000..0x2000 Phy 0x7307f9000..0x7307fa000 Flags 0
Log 0x2000..0x3000 Phy 0x731078000..0x731079000 Flags 0
Log 0x3000..0x5000 Phy 0x73127d000..0x73127f000 Flags FIEMAP_EXTENT_ENCODED
Log 0x5000..0x6000 Phy 0x73137a000..0x73137b000 Flags 0
Log 0x6000..0x7000 Phy 0x731683000..0x731684000 Flags 0
Log 0x7000..0x8000 Phy 0x73224f000..0x732250000 Flags 0
Log 0x8000..0x9000 Phy 0x7323c9000..0x7323ca000 Flags 0
Log 0x9000..0xb000 Phy 0x732425000..0x732427000 Flags FIEMAP_EXTENT_ENCODED
Log 0xb000..0xc000 Phy 0x732598000..0x732599000 Flags 0
Log 0xc000..0xd000 Phy 0x7325d5000..0x7325d6000 Flags FIEMAP_EXTENT_LAST
# fiewalk foo
exception type std::system_error: BTRFS_IOC_TREE_SEARCH_V2: /tmp/foo at fs.cc:844: Value too large for defined data type
Normally crawlers simply skip over inline extents, but ExtentWalker will
seek backward from the first non-inline extent to confirm that it has
an accurate starting block for the target extent. This fails when it
encounters the first inline extent.
strace reveals that buffer size is too small for the first extent,
as seen here:
ioctl(3, BTRFS_IOC_TREE_SEARCH_V2, {key={tree_id=258, min_objectid=78897856, max_objectid=UINT64_MAX, min_offset=0, max_offset=UINT64_MAX, min_transid=0, max_transid=UINT64_MAX, min_type=BTRFS_EXTENT_DATA_KEY, max_type=BTRFS_EXTENT_DATA_KEY, nr_items=16}, buf_size=1360} => {buf_size=1418}) = -1 EOVERFLOW (Value too large for defined data type)
Fix this by increasing the buffer size until it can handle the largest
possible object on the largest possible btrfs metadata page (65536 bytes).
BtrfsExtentWalker already has optimizations to minimize the allocation
cost, so we don't need any changes there.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
This is especially useful when dynamic load management allocates more
worker threads than active tasks, so the extra threads are effectively
invisible.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
Enable much simpler Task management: each time a Task needs to be done
at least once in the future, simply invoke the run() method on the Task.
The Task will ensure that it only runs once, only appears in a queue
once, and will run again if a run request is made while the Task is
already running.
Make the queue policy a member of the Task rather than a method. This
enables Tasks to reschedule themselves, possibly on the appropriate queue
if we have more than one of those some day.
This happens to make Tasks more similar to Linux kernel workers.
This similarity is coincidental, but not undesirable.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
SIGUNUSED is not defined in many environments (it seems to be defined
in only one I've tried so far). Hide the reference with #ifdef.
Fixes: https://github.com/Zygo/bees/issues/94
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
libcrucible at one time in the distant past had to be a shared library
to force global C++ object initialization; however, this is no longer
required.
Make libcrucible static to solve various rpath and soname versioning
issues, especially when distros try (unwisely) to package the library
separately.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
We need to replace nanosleeps with condition variables so that we
can implement BeesContext::stop. Export the time calculation from
sleep_for() into a new method called sleep_time().
If the thread executing RateLimiter::sleep_for() is interrupted, it will
no longer be able to restart, as the sleep_time() method is destructive.
This calls for further refactoring of sleep_time() into destructive
and non-destructive parts; however, there are currently no users of
sleep_for() which rely on being able to restart after being interrupted
by a signal.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
Add a method to have TaskMaster discard any entries in its queue, terminate
all worker threads, and prevent any new Tasks from being queued.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
We didn't take enough care to fix all invocations of git in this
scenario.
Fixes: 32d2739 ("Makefile: Specify version when building from tarball")
Signed-off-by: Kai Krakow <kai@kaishome.de>
Make sure the result set is empty before running the ioctl in case
something tries to consume the result without checking the error status.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
If we are not zero-filling containers then the overhead of allocating them
on each use is negligible. The effect that the thread_local containers
were having on RAM usage was very non-negligible.
Use dynamic containers (members or stack objects) for better control
of object lifetimes and much lower peak RAM usage. They're a tiny bit
faster, too.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
This commit brings back -O3 but in an overridable way. This should make
downstream distributions happy enough to accept it.
While at the subject, let's apply the same fixup logic to LDFLAGS, too.
This commit also properly gets rid of the implicit rules which collided
too easily with the depends.mk.
Signed-off-by: Kai Krakow <kai@kaishome.de>
Automatically fall back to LOGICAL_INO if LOGICAL_INO_V2 fails and no
_V2 flags are used.
Add methods to set the flags argument with build portability to older
headers.
Use thread_local storage for the somewhat large buffers used by
LOGICAL_INO_V2 (and other users of BtrfsDataContainer like INO_PATHS).
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
The -g option limits the number of worker threads when the target load
average is exceeded. On some systems the load normally runs high, and
continuous bees operation is required to avoid running out of disk space.
Add a -G/--thread-min option to force at least some threads to continue
running.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
Add -g / --loadavg-target parameter to track system load and add or
remove bees worker threads dynamically to keep system load close to the
loadavg target. Thread count may vary from zero to the maximum
specified by -c or -C, and is adjusted every 5 seconds.
This is better than implementing a similar load average scheme from
outside of the process (though that is still possible) because the
in-process load tracker does not disrupt the performance timing feedback
mechanisms as a freezer cgroup or SIGSTOP would when controlling bees
from outside. The internal load average tracker can also adjust the
number of active threads while an external tracker can only choose from
the maximum or zero.
Also fix a bug where a Task could deadlock waiting for itself to exit
if it tries to insert a new Task after the number of worker threads has
been set to zero.
Also correct usage message for --scan-mode (values are 0..2) since
we are touching adjacent lines anyway.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
Due to VPATH and how make resolves source paths, libcrucible.so ends up
with a hard-coded path to link against libuuid.so. Let's fix it by
turning the general rule into an explicit rule for libcrucible.so.
Signed-off-by: Kai Krakow <kai@kaishome.de>
We should probably not put it into the objects list. Let's instead
explicitly put it as a depend of libcrucible.so.
This allows us to not use *.cc as a depend for .version.cc which makes
more sense as CRUCIBLE_OBJS is also explicitly defined and not built
from wildcards.
Signed-off-by: Kai Krakow <kai@kaishome.de>
This commit adds support for putting package configuration options into
header files. This is needed to prepare reading config files from /etc.
Signed-off-by: Kai Krakow <kai@kaishome.de>
The memset is just doing an assignment from one dereferenced pointer to
another, so do an assignment to keep GCC 8 happy.
Fixes: https://github.com/Zygo/bees/issues/64
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
Extents that extend past EOF will have ipos = (file size rounded up
to next block) and e.end() = (file size not rounded), which fails this
constraint check.
The constraint check is wrong. Remove it for now.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
update_monotonic does not reset the counter if a new count is smaller than
earlier counts. Useful when consuming an unsorted stream of eveent counts.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
RateEstimator estimates the rate of external events by sampling a
counter.
Conversion functions are provided to predict the time when the
event counter will be incremented to particular values based on past
observations of the event counter.
Synchronization functions are provided to block a thread until a specific
counter value is reached.
Event polling is supported using the history of previous event counts
to determine the predicted time of the next event. A decay function
emphasizes more recent event history.
Polling delays are bounded by minimum and maximum values in the constructor
parameters.
wait_for() and wait_until() block the calling thread until the target
event count is reached (or the counter is reset). These functions are
not bounded by min_delay or max_delay, and require a separate tread
to call update(). wait_for() waits for the counter to be incremented
from its current value by the given count. wait_until() waits for the
counter to reach an absolute value.
update() counts external events and unblocks threads that are blocked
in wait_for() or wait_until(). If the event counter decreases then it
is reset to the new value.
duration() and time_point() convert relative and absolute event counts
into relative and absolute C++11 time quantities based on the last update
time, last observed event count, and the observed event rate.
Convenience functions seconds_for() and seconds_until() calculate
polling delays for for the desired relative and absolute event counts
respectively. These delays are bounded by max and min delay parameters.
rate() and ratio() provide conversion factors based on the current
estimated event rate.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
Since we are now unconditionally rendering the print_fn as a static
string, there is no need for it to be a function. We also need it to
be brief and mostly constant.
Use a string instead. Put the string before the function in the Task
constructor arguments so that the title string appears as a heading in
code, since we are making a breaking API change already.
Drop TASK_MACRO as it is broken by this change, but there is no similar
usage of Task anywhere to make it worth fixing.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
Move pthread_setname_np to the same place we do pthread_getname_np.
Detect errors in pthread_getname_np--but don't throw an exception
because we would call ourself recursively from the exception handler
when it tries to log the exception.
Fix the order of set_name and the first BEESNOTE/BEESLOG call in threads,
closing small time intervals where logs have the wrong thread name,
and that wrong name becomes persistent for the thread.
Make the main thread's name "bees" because Linux kernel stack traces use
the pthread name of the main thread instead of the name of the process.
Anonymous threads get the process name (usually "bees"). We should not
have any such threads, but we do. This appears to occur mostly during
exception stack unwinding. GCC/pthread bug?
Fixes: https://github.com/Zygo/bees/issues/51
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
When timestamps are removed from logging, the current text layout shows
lines like
tid 12345 thread_name: Example log
Let's convert it to a more conforming layout:
thread_name[12345]: Example log
Signed-off-by: Kai Krakow <kai@kaishome.de>
This enables bees' thread introspection to use task descriptions in
status and log messages.
BeesNote will be calling Task::current_task() from non-Task contexts,
which means we need to allow Task's shared state pointer to be null.
Remove some asserts that will ruin our day in that case.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
