docs: reorganize options, add workaround for btrfs send

options.md was a disorganized mess that markdown couldn't parse properly.

Break the options list down into sections by theme.  Add the new
'--workaround-btrfs-send' option to the new 'Workarounds' section.

Clean up the rest of the text and fix some inconsistencies.

Signed-off-by: Zygo Blaxell <bees@furryterror.org>

docs/btrfs-kernel.md

@@ -38,9 +38,6 @@ Unfixed kernel bugs (as of 4.14.71):
   `rsync` is copying from, while `rsync` will rename the new file over
   the old file to replace it.
 
-* **btrfs send** has various problems when bees is deduping RO snapshots,
-  especially if the snapshot is used as a parent for incremental send.
-
 Minor kernel problems with workarounds:
 
 * **Slow backrefs** (aka toxic extents):  Under certain conditions,
@@ -52,6 +49,19 @@ Minor kernel problems with workarounds:
   to get slow.  In the bees log, such blocks are labelled as 'toxic'
   hash/block addresses.
 
+* **btrfs send** has various bugs that are triggered when bees is
+  deduping snapshots.  bees provides the [`--workaround-btrfs-send`
+  option](options.md) which should be used whenever `btrfs send` and
+  bees are run on the same filesystem.
+
+  This issue affects:
+   * `btrfs send` (any mode) and bees active at the same time.
+   * `btrfs send` in incremental mode (using `-p` option) with bees
+     active at the same or different times.
+
+  Note `btrfs receive` is not affected.  It is OK to run bees with no
+  workarounds on a filesystem that receives btrfs snapshots.
+
 Older kernels:
 
 * Older kernels have various data corruption and deadlock/hang issues

docs/btrfs-other.md

@@ -28,11 +28,9 @@ bees has been tested in combination with the following, and various problems are
 * bcache, lvmcache:  **severe (filesystem-destroying) metadata corruption
   issues** observed in testing and reported by users, apparently only when
   used with bees.  Plain SSD and HDD seem to be OK.
-* btrfs send:  some kernel versions have bugs in btrfs send that can be
+* btrfs send:  there are bugs in `btrfs send` that can be triggered by bees.
-  triggered by bees.  The send can be restarted and will work if bees
+  The [`--workaround-btrfs-send` option](options.md) works around this issue,
-  has finished processing the snapshot being sent.  No data corruption
+  but possibly at great cost.
-  observed other than the truncated send.  Incremental send doesn't seem
-  to work with bees running on the sending side.
 * btrfs qgroups:  very slow, sometimes hangs...and it's even worse when
   bees is running.
 * btrfs autodefrag mount option:  hangs and high CPU usage problems

docs/options.md

@@ -1,52 +1,92 @@
 # bees Command Line Options
 
-<table border>
+## Load management options
-<tr><th width="20%">--thread-count COUNT</th><th width="5%">-c</th>
-<td>Specify maximum number of worker threads for scanning.  Overrides
---thread-factor (-C) and default/autodetected values.
-</td></tr>
-<tr><th>--thread-factor FACTOR</th><th>-C</th>
-<td>Specify ratio of worker threads to CPU cores.  Overridden by --thread-count (-c).
-    Default is 1.0, i.e. 1 worker thread per detected CPU.  Use values
-    below 1.0 to leave some cores idle, or above 1.0 if there are more
-    disks than CPUs in the filesystem.
-</td></tr>
 
-<tr><th>--loadavg-target LOADAVG</th><th>-g</th>
+* `--thread-count COUNT` or `-c`
-  <td>Specify load average target for dynamic worker threads.
-    Threads will be started or stopped subject to the upper limit imposed
-    by thread-factor, thread-min and thread-count until the load average
-    is within +/- 0.5 of LOADAVG.
-</td></tr>
-<tr><th>--thread-min COUNT</th><th>-G</th>
-<td>Specify minimum number of worker threads for scanning.
-    Ignored unless -g option is used to specify a target load.</td></tr>
 
-<tr><th>--scan-mode MODE</th><th>-m</th>
+ Specify maximum number of worker threads.  Overrides `--thread-factor`
-  <td>
+ (`-C`) and default/autodetected values.
-Specify extent scanning algorithm.  Default mode is 0.
+
-<em>EXPERIMENTAL</em> feature that may go away.
+* `--thread-factor FACTOR` or `-C`
-<ul>
+
-<li> Mode 0: scan extents in ascending order of (inode, subvol, offset).
+ Specify ratio of worker threads to detected CPU cores.  Overridden by
+ `--thread-count` (`-c`).
+
+ Default is 1.0, i.e. 1 worker thread per detected CPU.  Use values
+ below 1.0 to leave some cores idle, or above 1.0 if there are more
+ disks than CPUs in the filesystem.
+
+* `--loadavg-target LOADAVG` or `-g`
+
+ Specify load average target for dynamic worker threads.  Default is
+ to run the maximum number of worker threads all the time.
+
+ Worker threads will be started or stopped subject to the upper limit
+ imposed by `--thread-factor`, `--thread-min` and `--thread-count`
+ until the load average is within +/- 0.5 of `LOADAVG`.
+
+* `--thread-min COUNT` or `-G`
+
+ Specify minimum number of dynamic worker threads.  This can be used
+ to force a minimum number of threads to continue running while using
+ `--loadavg-target` to manage load.
+
+ Default is 0, i.e. all bees worker threads will stop when the system
+ load exceeds the target.
+
+ Has no effect unless `--loadavg-target` is used to specify a target load.
+
+## Filesystem tree traversal options
+
+* `--scan-mode MODE` or `-m`
+
+ Specify extent scanning algorithm.  Default `MODE` is 0.
+ **EXPERIMENTAL** feature that may go away.
+
+  * Mode 0: scan extents in ascending order of (inode, subvol, offset).
   Keeps shared extents between snapshots together.  Reads files sequentially.
-Minimizes temporary space usage.</li>
+  Minimizes temporary space usage.
-<li> Mode 1: scan extents from all subvols in parallel.  Good performance
+  * Mode 1: scan extents from all subvols in parallel.  Good performance
-  on non-spinning media when subvols are unrelated.</li>
+  on non-spinning media when subvols are unrelated.
-<li> Mode 2: scan all extents from one subvol at a time.  Good sequential
+  * Mode 2: scan all extents from one subvol at a time.  Good sequential
-  read performance for spinning media.  Maximizes temporary space usage.</li>
+  read performance for spinning media.  Maximizes temporary space usage.
-</ul>
-</td></tr>
 
-<tr><th>--timestamps</th><th>-t</th>
+## Workarounds
-  <td>Enable timestamps in log output.</td></tr>
-<tr><th>--no-timestamps</th><th>-T</th>
-  <td>Disable timestamps in log output.</td></tr>
-<tr><th>--absolute-paths</th><th>-p</th>
-  <td>Paths in log output will be absolute.</td></tr>
-<tr><th>--strip-paths</th><th>-P</th>
-  <td>Paths in log output will have the working directory at bees startup
-    stripped.</td></tr>
-<tr><th>--verbose</th><th>-v</th>
-  <td>Set log verbosity (0 = no output, 8 = all output, default 8).</td></tr>
 
-</table>
+* `--workaround-btrfs-send` or `-a`
+
+ Pretend that read-only snapshots are empty and silently discard any
+request to dedupe files referenced through them.  This is a workaround for
+[problems with the kernel implementation of `btrfs send` and `btrfs send
+-p`](btrfs-kernel.md) which make these btrfs features unusable with bees.
+
+ This option should be used to avoid breaking `btrfs send` on the same
+filesystem.
+
+ **Note:** There is a _significant_ space tradeoff when using this option:
+it is likely no space will be recovered--and possibly significant extra
+space used--until the read-only snapshots are deleted.  On the other
+hand, if snapshots are rotated frequently then bees will spend less time
+scanning them.
+
+## Logging options
+
+* `--timestamps` or `-t`
+
+ Enable timestamps in log output.
+
+* `--no-timestamps` or `-T`
+
+ Disable timestamps in log output.
+
+* `--absolute-paths` or `-p`
+
+ Paths in log output will be absolute.
+
+* `--strip-paths` or `-P`
+
+ Paths in log output will have the working directory at bees startup stripped.
+
+* `--verbose` or `-v`
+
+ Set log verbosity (0 = no output, 8 = all output, default 8).