GlusterFS: "failed to exec fusermount"

[peter_b]

[PROBLEM]
I've had our glusterfs server mounted on Debian 7 without problems. After unmounting it, and changing some network configuration settings on the Gluster server (v3.4.6, RHEL 6.5), I suddenly get the following errors in /var/log/glusterfs/ logs:

0-glusterfs-fuse: failed to exec fusermount: No such file or directory

The whole block reads:

[2015-03-16 13:52:18.511960] W [socket.c:582:__socket_rwv] 0-dlp-storage-client-0: readv failed (No data available)
[2015-03-16 13:52:18.512909] W [socket.c:582:__socket_rwv] 0-dlp-storage-client-1: readv failed (No data available)
[2015-03-16 13:52:18.513734] W [socket.c:582:__socket_rwv] 0-dlp-storage-client-3: readv failed (No data available)
[2015-03-16 13:52:18.514435] W [socket.c:582:__socket_rwv] 0-dlp-storage-client-4: readv failed (No data available)
[2015-03-16 13:52:18.515048] W [socket.c:582:__socket_rwv] 0-dlp-storage-client-5: readv failed (No data available)
[2015-03-16 13:52:18.515868] W [socket.c:582:__socket_rwv] 0-dlp-storage-client-2: readv failed (No data available)
[2015-03-16 13:52:18.529048] E [mount.c:162:fuse_mount_fusermount] 0-glusterfs-fuse: failed to exec fusermount: No such file or directory
[2015-03-16 13:52:18.529676] E [mount.c:298:gf_fuse_mount] 0-glusterfs-fuse: mount of vm-gluster-1.bb2.dlp-storage.local:/dlp-storage to /mnt/dlp-storage (default_permissions,noatime,allow_other,max_read=131072) failed
[2015-03-16 13:52:18.530382] E [glusterfsd.cdaemonize] 0-daemonize: mount failed
[2015-03-16 13:52:18.531169] W [glusterfsd.ccleanup_and_exit] (-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f0ce3b3a7bd] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7f0ce41e8b50] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xd5) [0x7f0ce4cda545]))) 0-: received signum (15), shutting down

I can however mount the gluster share without problems directly on the RHEL6 machines. Additionally, another Debian7 client which did *not* have the share unmounted for the changes, can still access it fine.

I tried restarting the glusterd on one of the GlusterFS nodes, but then I just got the following additional error messages on the Debian client side:

[2015-03-16 14:23:09.480087] W [fuse-bridge.c:4576:fuse_thread_proc] 0-glusterfs-fuse: read from /dev/fuse returned -1 (Operation not permitted)

So I checked the permission of "/dev/fuse".
There's something fishy going on here.

On the Debian client that's broken, it's:

Code: Select all

crw------T 1 root root 10, 229 Dec 18 14:46 /dev/fuse

On a working Debian client it's:

Code: Select all

crw-rw---T 1 root fuse 10, 229 Dec 18 14:45 /dev/fuse

Strange, but I've tried manually fixing the broken client's /dev/fuse permissions: nothing changed

[SOLUTION]
Damn, this one was tricky...
The option "noatime" in fstab is causing the problem! I've obviously added it, but never tried it (umount/remount) - so I thought it worked before.

Seems like it never did:
See the "[enhancement]: Add support for noatime, nodiratime" (Redhat Bug #825569) ticket for details.

As is mentioned in the request ticket:
If "noatime/nodiratime" options are desired, add them on the mount points of the bricks instead.