context: move TempFile from TLS to Pool and fix some FdCache issues

Get rid of the thread-local TempFiles and use Pool instead.  This
eliminates a potential FD leak when the loadavg governor repeatedly
creates and destroys threads.

With the old per-thread TempFiles, we were guaranteed to have exclusive
ownership of the TempFile object within the current thread.  Pool is
somewhat stricter:  it only guarantees ownership while the checked-out
Handle exists.  Adjust the users of TempFile objects to ensure they hold
the Handle object until they are finished using the TempFile.

It appears that maintaining large, heavily-reflinked, long-lived temporary
files costs more than truncating after every use: btrfs has to write
multiple references to the temporary file's extents, then some commits
later, remove references as the temporary file is deleted or truncated.
Using the temporary file in a dedupe operation flushes the data to disk,
so nothing is saved by pretending that there is writeback pipelining and
trying to avoid flushes in truncate.  Pool provides usage tracking and
a checkin callback, so use it to truncate the temporary file immediately
after every use.

Redesign TempFile so that every instance creates exactly one Fd which
persists over the lifetime of the TempFile object.  Provide a reset()
method which resets the file back to the initial state and call it from
the Pool checkin callback.  This makes TempFile's lifetime equivalent to
its Fd's lifetime, which simplifies interactions with FdCache and Roots.

This change means we can now blacklist temporary files without having
an effective memory leak, so do that.  We also have a reason to ever
remove something from the blacklist, so add a method for that too.

In order to move to extent-centric addressing, we need to be able to
reliably open temporary files by root and inode number.  Previously we
would place TempFile fd's into the cache with insert_root_ino, but the
cache would be cleared periodically, and it would not be possible to
reopen temporary files after that happened.  Now that the TempFile's
lifetime is the same as the TempFile Fd's lifetime, we can have TempFile
manage a separate FileId -> Fd map in Roots which is unaffected by the
periodic cache clearing.  BeesRoots::open_root_ino_nocache will check
this map before attempting to open the file via btrfs root+ino lookup,
and return it through the cache as if Roots had opened the file via btrfs.

Hold a reference to BeesRoots in BeesTempFile because the usual way
to get such a reference now throws an exception in BeesTempFile's
destructor.

These changes make method BeesTempFile::create() and all methods named
insert_root_ino unnecessary, so delete them.

We construct and destroy TempFiles much less often now, so make their
constructor and destructor more informative.

Signed-off-by: Zygo Blaxell <bees@furryterror.org>

src/bees.h

@@ -8,6 +8,7 @@
 #include "crucible/fd.h"
 #include "crucible/fs.h"
 #include "crucible/lockset.h"
+#include "crucible/pool.h"
 #include "crucible/progress.h"
 #include "crucible/time.h"
 #include "crucible/task.h"
@@ -546,6 +547,9 @@ class BeesRoots : public enable_shared_from_this<BeesRoots> {
 	bool					m_workaround_btrfs_send = false;
 	LRUCache<bool, uint64_t>		m_root_ro_cache;
 
+	mutex					m_tmpfiles_mutex;
+	map<BeesFileId, Fd>			m_tmpfiles;
+
 	mutex					m_stop_mutex;
 	condition_variable			m_stop_condvar;
 	bool					m_stop_requested = false;
@@ -572,9 +576,12 @@ class BeesRoots : public enable_shared_from_this<BeesRoots> {
 	RateEstimator& transid_re();
 	size_t crawl_batch(shared_ptr<BeesCrawl> crawl);
 	void clear_caches();
+	void insert_tmpfile(Fd fd);
+	void erase_tmpfile(Fd fd);
 
 friend class BeesFdCache;
 friend class BeesCrawl;
+friend class BeesTempFile;
 
 public:
 	BeesRoots(shared_ptr<BeesContext> ctx);
@@ -668,10 +675,10 @@ friend ostream & operator<<(ostream &os, const BeesRangePair &brp);
 
 class BeesTempFile {
 	shared_ptr<BeesContext> m_ctx;
+	shared_ptr<BeesRoots>   m_roots;
 	Fd			m_fd;
 	off_t			m_end_offset;
 
-	void create();
 	void realign();
 	void resize(off_t new_end_offset);
 
@@ -680,6 +687,7 @@ public:
 	BeesTempFile(shared_ptr<BeesContext> ctx);
 	BeesFileRange make_hole(off_t count);
 	BeesFileRange make_copy(const BeesFileRange &src);
+	void reset();
 };
 
 class BeesFdCache {
@@ -692,7 +700,6 @@ public:
 	BeesFdCache();
 	Fd open_root(shared_ptr<BeesContext> ctx, uint64_t root);
 	Fd open_root_ino(shared_ptr<BeesContext> ctx, uint64_t root, uint64_t ino);
-	void insert_root_ino(shared_ptr<BeesContext> ctx, Fd fd);
 	void clear();
 };
 
@@ -715,7 +722,7 @@ class BeesContext : public enable_shared_from_this<BeesContext> {
 	shared_ptr<BeesFdCache>				m_fd_cache;
 	shared_ptr<BeesHashTable>			m_hash_table;
 	shared_ptr<BeesRoots>				m_roots;
-	map<thread::id, shared_ptr<BeesTempFile>>	m_tmpfiles;
+	Pool<BeesTempFile>				m_tmpfile_pool;
 
 	LRUCache<BeesResolveAddrResult, BeesAddress>	m_resolve_cache;
 
@@ -763,10 +770,11 @@ public:
 	BeesFileRange scan_forward(const BeesFileRange &bfr);
 
 	bool is_root_ro(uint64_t root);
-	BeesRangePair dup_extent(const BeesFileRange &src);
+	BeesRangePair dup_extent(const BeesFileRange &src, const shared_ptr<BeesTempFile> &tmpfile);
 	bool dedup(const BeesRangePair &brp);
 
-	void blacklist_add(const BeesFileId &fid);
+	void blacklist_insert(const BeesFileId &fid);
+	void blacklist_erase(const BeesFileId &fid);
 	bool is_blacklisted(const BeesFileId &fid) const;
 
 	BeesResolveAddrResult resolve_addr(BeesAddress addr);
@@ -786,9 +794,6 @@ public:
 
 	const Timer &total_timer() const { return m_total_timer; }
 	LockSet<uint64_t> &extent_lock_set() { return m_extent_lock_set; }
-
-	// TODO: move the rest of the FD cache methods here
-	void insert_root_ino(Fd fd);
 };
 
 class BeesResolver {