{"id":986,"date":"2026-11-14T09:52:38","date_gmt":"2026-11-14T09:52:38","guid":{"rendered":"https:\/\/reversea.me\/?p=986"},"modified":"2026-11-14T09:54:13","modified_gmt":"2026-11-14T09:54:13","slug":"watch-out-for-your-watchers-how-a-file-watch-bug-froze-our-encrypted-folders-and-a-one-line-bug-in-macfuse","status":"publish","type":"post","link":"https:\/\/reversea.me\/index.php\/watch-out-for-your-watchers-how-a-file-watch-bug-froze-our-encrypted-folders-and-a-one-line-bug-in-macfuse\/","title":{"rendered":"Watch Out for Your Watchers: How a File-Watch Bug Froze Our Encrypted Folders (and a One-Line Bug in macFUSE)"},"content":{"rendered":"Reading Time: <\/span> 11<\/span> minutes<\/span><\/span>\n

TL;DR<\/strong> We write our papers in LaTeX, inside folders encrypted with EncFS<\/a> on top of macFUSE<\/a>, with VS Code rebuilding and refreshing the PDF preview on every save. One evening the encrypted volume froze so hard that even shutting down the Mac stalled, twice in fifteen minutes. In this post we walk through how we went from the macOS shutdown-stall reports to the root cause: since macFUSE 5.1.0, the libfuse<\/code> handler for the new FUSE_MONITOR<\/code> message takes a path lock and never releases it. A normal attach\/detach cycle for a per-file kqueue watch generates a MONITOR<\/code> BEGIN\/END pair, and the affected handler leaks one read lock for each message. A later FORGET<\/code> that drops the last lookup reference to a leaked node can then park a worker thread forever. An old accounting shortcut in libfuse<\/code> leaves blocked FORGET<\/code> workers counted as available, so enough such requests can exhaust the real worker pool without starting replacements. We show the bug in the source, confirm its affected release window in the binaries and source history, and reproduce the full freeze with a toy file system of a few hundred lines of C. The mechanism is demonstrated end to end; the shutdown-stall reports do not, however, identify which application created the original watches. The bug was already reported upstream (macfuse\/macfuse#1168<\/a>) and is fixed since macFUSE 5.3.1, so the short version is: update macFUSE<\/em>. The long version is below :).<\/p>\n\n\n\n

\n\n\n\n

How It Started: a .tex<\/code>, a PDF Preview, and a Frozen Mac<\/h2>\n\n\n\n

Our day-to-day paper writing is pretty boring, which is how we like it: .tex<\/code> sources, VS Code with the LaTeX Workshop<\/a> extension, latexmk<\/code> building on every save, and the PDF preview tab refreshing itself a second later. The only slightly unusual part is where<\/em> the sources live. Our research folders are encrypted with EncFS, and the ciphertext is synced to our own Nextcloud server (when it works). EncFS is a FUSE file system, so on macOS it runs on top of macFUSE: the kernel extension forwards operations that require user-space handling to the encfs<\/code> process, which decrypts on the fly.<\/p>\n\n\n\n

On November 5, 2026, in the middle of an evening of writing, everything that touched the encrypted folder stopped responding (this was happening us few weeks ago, but we did not have enough time to dig into it until now…). The editor hung, terminals hung on ls<\/code>, and diskutil unmount<\/code> did not help. We rebooted. Eleven minutes after remounting the folder and getting back to work, it happened again<\/em>. Second reboot.<\/p>\n\n\n\n

Two freezes in fifteen minutes is not bad luck, it is a bug. And since the Mac kept forensic breadcrumbs of both incidents, we went digging (well, spoiler here<\/em>: the culprit is a missing free_path()<\/code> call, otherwise you would not be reading this<\/strong>).<\/p>\n\n\n\n

First Clues: Reading the Shutdown-Stall Reports<\/h2>\n\n\n\n

When macOS takes too long to shut down, launchd<\/code> records a shutdown stall<\/em> report in \/Library\/Logs\/DiagnosticReports\/<\/code>. These reports are binary spindumps, but spindump<\/code> itself can turn them into text:<\/p>\n\n\n\n

spindump -i \/Library\/Logs\/DiagnosticReports\/shutdown_stall_2026-11-05-215306_atropos.shutdownStall -o stall1.txt<\/code><\/pre>\n\n\n\n
Each report contains the stack of every thread of every process still alive while the system was trying to shut down. Of course, the encfs<\/code> processes were there. Figure 1 shows the process serving our research folder: the main thread waits in the libfuse<\/code> session loop, as expected, while every worker thread sits on exactly the same stack. Note the last ran<\/code> column: one worker had not run for 38 hours, another for 21 hours, and the rest had stopped at the same instant, less than three minutes before the shutdown.<\/p>\n\n\n\n
\"Excerpt<\/span>
\n
<\/p>\n<\/div><\/div>\n\n\n\n
Figure 1: Excerpt of the decoded shutdown-stall report: the encfs<\/code> process serving our research folder (annotations ours).<\/em><\/p>\n\n\n\n
The library is stripped, but the arm64 slice of libfuse.2.dylib<\/code> disassembles nicely with otool -arch arm64 -tV<\/code>. The four libfuse<\/code> offsets in the worker stack (59684, 69888, 26228 and 48904, that is, 0xe924<\/code>, 0x11100<\/code>, 0x6674<\/code> and 0xbf08<\/code>) are, respectively, the worker loop right after fuse_session_process_buf()<\/code>, the request dispatcher, a small handler that prints \"FORGET %llu\/%llu\\n\"<\/code> in debug mode, and a pthread_cond_wait()<\/code> inside the function that handler calls. In other words: every worker is processing a FORGET<\/code> request and waiting on a condition variable<\/strong>.<\/p>\n\n\n\n
And it was not alone. Counting threads across both reports:<\/p>\n\n\n\n
Report<\/th>encfs<\/code> process<\/th>Workers waiting at 0xbf08<\/code><\/th>Workers reading \/dev\/macfuse<\/code><\/th>Notes<\/th><\/tr><\/thead>
21:52 (first freeze)<\/td>research folder<\/td>9 \/ 9<\/strong><\/td>0<\/td>2 of them had been parked for 38 h and 21 h; the other 7 since 21:50:01<\/td><\/tr>
21:52 (first freeze)<\/td>second encrypted folder<\/td>3 \/ 9<\/td>6<\/td>2 parked for ~37 h, 1 since 21:49: quietly heading the same way<\/td><\/tr>
22:09 (second freeze)<\/td>research folder, remounted<\/td>10 \/ 10<\/strong><\/td>0<\/td>all parked within 11 minutes of mounting<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
So the workers were not stuck on I\/O, on decryption, or on Nextcloud. They were waiting, one by one, for something that never came, and once the last one got in line, nobody was left to read requests from the kernel. Time to read some source code.<\/p>\n\n\n\n
Into the Source: Path Locks in High-Level libfuse<\/code><\/h2>\n\n\n\n
EncFS uses the high-level<\/em> FUSE API, where the file system receives paths (\/papers\/draft\/paper.pdf<\/code>) instead of inode numbers. To keep paths consistent while requests run concurrently, libfuse<\/code> maintains an in-memory tree of nodes and a per-node lock counter, treelock<\/code>: positive values count readers, -1<\/code> (TREELOCK_WRITE<\/code>) marks a writer. macFUSE maintains its own fork of libfuse<\/code> 2.9 in macfuse\/library<\/a> (branch support\/libfuse-2.9<\/code>), and the read-locking part of try_get_path()<\/code> in lib\/fuse.c<\/code> looks like this:<\/p>\n\n\n\n
for (node = get_node(f, nodeid); node->nodeid != FUSE_ROOT_ID;\n     node = node->parent) {\n\t...\n\tif (need_lock) {\n\t\terr = -EAGAIN;\n\t\tif (node->treelock < 0)\n\t\t\tgoto out_unlock;\n\n\t\tnode->treelock++;\n\t}\n}<\/code><\/pre>\n\n\n\n
Note the loop: a read lock on a path increments treelock<\/code> on the node and on every ancestor directory<\/strong> up to (but not including) the root. Every get_path()<\/code> must be paired with a free_path()<\/code>, which walks the same chain decrementing the counters.<\/p>\n\n\n\n
Now, the function our threads were stuck in. The FORGET<\/code> handler calls forget_node()<\/code>:<\/p>\n\n\n\n
static void forget_node(struct fuse *f, fuse_ino_t nodeid, uint64_t nlookup)\n{\n\t...\n\t\/*\n\t * Node may still be locked due to interrupt idiocy in open,\n\t * create and opendir\n\t *\/\n\twhile (node->nlookup == nlookup && node->treelock) {\n\t\tstruct lock_queue_element qe = {\n\t\t\t.nodeid1 = nodeid,\n\t\t};\n\n\t\tdebug_path(f, \"QUEUE PATH (forget)\", nodeid, NULL, false);\n\t\tqueue_path(f, &qe);\n\n\t\tdo {\n\t\t\tpthread_cond_wait(&qe.cond, &f->lock);\n\t\t} while (node->nlookup == nlookup && node->treelock);\n\t\t...<\/code><\/pre>\n\n\n\n
When the kernel asks the file system to drop the last lookup reference<\/em> known to libfuse<\/code> (node->nlookup == nlookup<\/code>), forget_node()<\/code> waits until nobody holds a path lock on that node. That pthread_cond_wait()<\/code> is our 0xbf08<\/code>. If treelock<\/code> never goes back to zero, the thread never comes back. So the question becomes: who is leaking path locks?<\/strong><\/p>\n\n\n\n
The Bug: a MONITOR<\/code> That Never Lets Go<\/h2>\n\n\n\n
macFUSE 5.1.0 (October 30, 2025) added support for a new macOS-specific message, FUSE_MONITOR<\/code> (opcode 60), in commit 67aa41ee<\/code><\/a>. The kernel sends it to tell the file system that the number of watchers of a file has changed (FUSE_MONITOR_BEGIN<\/code> \/ FUSE_MONITOR_END<\/code>). Here is the high-level handler as it shipped:<\/p>\n\n\n\n
static void fuse_lib_monitor(fuse_req_t req, fuse_ino_t ino, uint32_t flags)\n{\n\tstruct fuse *f = req_fuse_prepare(req);\n\tchar *path;\n\tint err;\n\n\terr = get_path(f, ino, &path);   \/* read lock on the node and its ancestors *\/\n\tif (err) {\n\t\tfuse_reply_none(req);\n\t\treturn;\n\t}\n\n\tfuse_fs_monitor(f->fs, path, flags);\n\tfuse_reply_none(req);            \/* ...and no free_path(). Ever. *\/\n}<\/code><\/pre>\n\n\n\n
There is a get_path()<\/code> but no free_path()<\/code>. Every MONITOR<\/code> message leaves treelock<\/code> incremented by one on the watched node and on each parent directory up to, but not including, the root. Two details make this worse than it looks:<\/p>\n\n\n\n