Tests could now be run in parallel. Additionally, single tests can be
run by simply using "make testname", i.e. "make chatter" would run the
chatter test.
Signed-off-by: Kai Krakow <kai@kaishome.de>
According to gcc docs, -l is converted to a filename which makes it a
filename parameter. Let's move it to the end.
Signed-off-by: Kai Krakow <kai@kaishome.de>
When timestamps are removed from logging, the current text layout shows
lines like
tid 12345 thread_name: Example log
Let's convert it to a more conforming layout:
thread_name[12345]: Example log
Signed-off-by: Kai Krakow <kai@kaishome.de>
When a Task worker thread is executing a Task, the thread name is less
useful than the Task description.
Use the Task description instead of the thread name if the thread has
no BeesThread name and the thread is currently executing a task.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
Threads from the Task module in libcrucible don't set BeesNote::tl_name.
Even if they did, in Task context the thread name is unspecific to the point
of meaninglessness.
Use the Task::print method as the name for such threads, and be sure
that future Task print functions are designed for that usage.
The extra complexity in BeesNote::get_name() seems preferable to
bombarding pthread_setname_np hundreds or thousands of times per second.
FIXME: we are now calling Task::print() on every BeesNote, which
is effectively unconditionally. Maybe we should have Task::print()
and get_name() return a closure, or just evaluate Task::print() once
and cache it in TaskState, or define Task's constructor with a string
argument instead of the current print_fn closure.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
This enables bees' thread introspection to use task descriptions in
status and log messages.
BeesNote will be calling Task::current_task() from non-Task contexts,
which means we need to allow Task's shared state pointer to be null.
Remove some asserts that will ruin our day in that case.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
Silence the three(!) log messages per crawl increment an extra one at
the end of the subvol.
The three critical messages per subvol crawl cycle are:
Next transid in BeesCrawlState <SUBVOL>:0 offset 0x0 transid <A>..<B> started <T> (<AGO>s ago)
Subvol has been completely scanned and a new transaction range will
be created. CrawlState is the state of the old subvol.
Restarted crawl BeesCrawlState <SUBVOL>:0 offset 0x0 transid <B>..<C> started <T+AGO> (0s ago)
Subvol has been restarted. CRawlState is the state of the new subvol.
Deferring next transid in BeesCrawlState <SUBVOL>:0 offset 0x0 transid <B>..<C> started <T+AGO> (0s ago)
Subvol has been completely scanned, but it is too soon to start a
new scan.
Fix the "Restart..." message to use the correct verb tense and to use
the correct BeesCrawlState data.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
When we find a matching block we attempt to extend ("grow") the matched
pair around the first matching block. This function takes the IO hit of
reading the second extent from each duplicate extent pair. It's also
very slow--too many allocations, too small reads, reads in the wrong
order, an order of magnitude too many calls to TREE_SEARCH_V2, and it
is usually in the top 3 most frequent PERFORMANCE warnings.
Start tracking the running time of grows using the pairforward_ms
and pairbackward_ms counters so that we can compare it to various
replacements.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
This commit adds log levels to the output. In systemd, it makes colored
lines, otherwise it's probably just a number. Bees is very chatty, so
this paves the road for log level filtering.
Signed-off-by: Kai Krakow <kai@kaishome.de>
Dependencies can be generated in parallel which can be much faster. It
also puts away the problem that for may fail multiple times in a row and
leaving behind a broken intermediate file which would be picked up by
successive runs.
Signed-off-by: Kai Krakow <kai@kaishome.de>
Let's generalize the depends.mk target so we can easily move files
around later. While doing it, let's also fix the "gcc -M" call to use
explicit target names and not clobber it with preprocessor output.
Signed-off-by: Kai Krakow <kai@kaishome.de>
We can remove the explicit depend on the .h file because that is covered
by depends.mk. Let's instead depend on makeflags which makes more sense.
Signed-off-by: Kai Krakow <kai@kaishome.de>
We need a better cache expiration algorithm than "make a copy of
the entire thing, sort it while holding a lock, and delete half
the items in a single burst."
Replace the Lamport clock with a double-linked list. Each insert
or lookup operation moves the affected item to the head of the list.
Each erase operation deletes one single item at the tail of the list.
Also sort out some iterator invalidation nonsense by doing erases before
inserts instead of "insert, erase, find the inserted item again because
we invalidated the found iterator during the erase."
The new implementation adds a second word-sized member to each Value
as well as a copy of the Key. Hopefully the enlarged size is not
a deal-breaker.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
The mlock runs much faster, probably because the hash fetches are
doing most of the work that mlock does.
It makes bees startup latency for testing smaller, even if it takes more
time in absolute terms.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
Include a brief description of the two algorithms without getting
into too much detail for an ostensibly temporary feature.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
There are two subvol scan algorithms implemented so far. The two modes
are unimaginatively named 0 and 1.
0: sorts extents by (inode, subvol, offset),
1: scans extents round-robin from all subvols.
Algorithm 0 scans references to the same extent at close to the same
time, which is good for performance; however, whenever a snapshot is
created, the scan of the entire filesystem restarts at the beginning of
the new snapshot.
Algorithm 1 makes continuous forward progress even when new snapshots
are created, but it does not benefit from caching and will force the
kernel to reread data multiple times when there are snapshots.
The algorithm can be selected at run-time using the -m or --scan-mode
option.
We can collect some field data on these before replacing them with
an extent-tree-based scanner. Alternatively, for pre-4.14 kernels,
we can keep these two modes as non-default options.
Currently these algorithms have terrible names. TODO: fix that, but
also TODO: delete all that code and do scans directly from the extent
tree instead.
Augment the scan algorithms relative to their earlier implementation by
batching multiple extents to scan from each subvol before switching to
a different subvol.
Sprinkle some BEESNOTEs on the Task objects so that they don't
disappear from the thread status output.
Adjust some timing constants to deal with the increased latency from
competing threads.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
Distribute incoming extents across a thread pool for faster execution
on multi-core, multi-disk environments.
Switch extent enumeration model to scan extent refs consecutively(ish).
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
In both instances the code contained within (or the conditional
compilation surrounding it) is no longer controversial.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
We need a mechanism for distributing work across processor cores and
disks.
Task implements a simple FIFO/LIFO queue model for executing closures.
Some locking primitives are included (mutex and barrier).
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
Bees will someday rely on features available only in kernel v4.14.
Let's start now by removing workarounds for bugs that were fixed in v4.11.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
Remove some dead code because dedup-related deadlocks have not been
observed since Linux kernel v4.11.
Preserve rationale of remaining #if 0 block (why we do write/rename
instead of write/fsync/rename) so that people don't try to replace the
"missing" fsync() there.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
With kernel 4.14 there is no sign of the previous LOGICAL_INO performance
problems, so there seems to be no need to throttle threads using this
ioctl.
Increase the FD cache size limits and scan thread count. Let the kernel
figure out scheduling.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
BEESNOTE puts a message on the status message stack. BEESINFO logs a
message with rate limiting. The message that was flooding the logs
was coming from BEESINFO not BEESNOTE.
Fix earlier commit which removed the wrong message.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
We were holding weak refs until the next time the resource ID was used.
This is a bad thing if resource IDs are sparse (e.g. pointers or hashes)
because we'll never see an ID twice.
To fix, determine whether we released the last instance of a resource,
and if so, free its weak ref immediately.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
The bugs in other parts of the code have been identified and fixed,
so the overprotective locks around shared_ptr can be removed.
Keep the other improvements to the Resource class.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
This avoids PERFORMANCE warnings when large hash tables are used on slow
CPUs or with lots of worker threads. It also simplifies the code (no
locksets, only one object-wide mutex instead of two).
Fixed a few minor bugs along the way (e.g. we were not setting the dirty
flag on the right hash table extent when we detected hash table errors).
Simplified error handling: IO errors on the hash table are ignored,
instead of throwing an exception into the function that tried to use the
hash table.
Signed-off-by: Zygo Blaxell <bees@furryterror.org>
This commit adds an ebuild for Gentoo. Version 9999 is building live
from current git, currently using kakra:integration because it has some
installation and build fixes important for Gentoo.
Signed-off-by: Kai Krakow <kai@kaishome.de>
According to Gentoo packaging guide, -fPIC should only be used on shared
libraries, and not added unconditionally to every linker call.
Signed-off-by: Kai Krakow <kai@kaishome.de>
