mirror of
https://github.com/Zygo/bees.git
synced 2025-05-18 05:45:45 +02:00
82474b4ef4
At least one user was significantly confused by "designed for large filesystems". The btrfs send workarounds aren't new any more. Signed-off-by: Zygo Blaxell <bees@furryterror.org>
77 lines
2.5 KiB
Markdown
77 lines
2.5 KiB
Markdown
BEES
|
|
====
|
|
|
|
Best-Effort Extent-Same, a btrfs deduplication agent.
|
|
|
|
About bees
|
|
----------
|
|
|
|
bees is a block-oriented userspace deduplication agent designed to scale
|
|
up to large btrfs filesystems. It is a daemon that performs offline
|
|
dedupe automatically as required. It uses an incremental data scan
|
|
capability to minimize memory usage and dedupe new data soon after it
|
|
appears in the filesystem.
|
|
|
|
Strengths
|
|
---------
|
|
|
|
* Space-efficient hash table and matching algorithms - can use as little as 1 GB hash table per 10 TB unique data (0.1GB/TB)
|
|
* Daemon incrementally dedupes new data using btrfs tree search
|
|
* Works with btrfs compression - dedupe any combination of compressed and uncompressed files
|
|
* Works around btrfs filesystem structure issues to free more disk space than generic dedupe tools
|
|
* Persistent hash table and checkpoint for rapid restart after shutdown
|
|
* Whole-filesystem dedupe - including snapshots
|
|
* Constant hash table size - no increased RAM usage if data set becomes larger
|
|
* Works on live data - no scheduled downtime required
|
|
* Automatic self-throttling based on system load
|
|
* Low memory footprint (excluding the hash table)
|
|
|
|
Weaknesses
|
|
----------
|
|
|
|
* Whole-filesystem dedupe - has no include/exclude filters, does not accept file lists, terminates only when explicitly requested
|
|
* Requires root privilege (or `CAP_SYS_ADMIN`) to work
|
|
* First run may require temporary disk space for extent reorganization
|
|
* [First run may increase metadata space usage if many snapshots exist](gotchas.md)
|
|
* Constant hash table size - no decreased RAM usage if data set becomes smaller
|
|
* btrfs only (bcachefs and xfs are missing various features)
|
|
|
|
Installation and Usage
|
|
----------------------
|
|
|
|
* [Installation](install.md)
|
|
* [Configuration](config.md)
|
|
* [Running](running.md)
|
|
* [Command Line Options](options.md)
|
|
|
|
Recommended Reading
|
|
-------------------
|
|
|
|
* [bees Gotchas](gotchas.md)
|
|
* [btrfs kernel bugs](btrfs-kernel.md) - especially DATA CORRUPTION WARNING
|
|
* [bees vs. other btrfs features](btrfs-other.md)
|
|
* [What to do when something goes wrong](wrong.md)
|
|
|
|
More Information
|
|
----------------
|
|
|
|
* [How bees works](how-it-works.md)
|
|
* [Missing bees features](missing.md)
|
|
* [Event counter descriptions](event-counters.md)
|
|
|
|
Bug Reports and Contributions
|
|
-----------------------------
|
|
|
|
Email bug reports and patches to Zygo Blaxell <bees@furryterror.org>.
|
|
|
|
You can also use Github:
|
|
|
|
https://github.com/Zygo/bees
|
|
|
|
Copyright & License
|
|
-------------------
|
|
|
|
Copyright 2015-2023 Zygo Blaxell <bees@furryterror.org>.
|
|
|
|
GPL (version 3 or later).
|