Mount fails when adding new node to GFS2 EBS multi-attached Clustered storage

Question

Hello,

We have setup EBS multi-attach clustered storage as per https://aws.amazon.com/blogs/storage/clustered-storage-simplified-gfs2-on-amazon-ebs-multi-attach-enabled-volumes/ and it is working fine on two EC2 nodes having centos 7 OS.

Now we have added 3rd EC2 node to GFS2 EBS Clustered storage, it is getting added fine via  command 
```
pcs cluster node add  --start --enable
```
 however mounting of EBS volume fails with error in log as:

```
Failed Resource Actions:
* xxx_data_cluster_start_0 on xxx-golan 'unknown error' (1): call=30, status=complete, exitreason='Couldn't mount device [/dev/xxvg/xxlv] as /data',
    last-rc-change='Tue Nov  8 10:56:47 2022', queued=0ms, exec=2272ms
```
When we manually try to mount EBS multi-attach volume from new node it gives error:

```
mount: mount /dev/mapper/xxxvg-xxxlv on /data failed: Too many users
```

What could be the problem ?

Thanks

Answer

"Too many users" would mean you don't have enough journals in your GFS2 filesystem.

> GFS2 requires one journal for each node in the cluster that needs to mount the file system. For example, if you have a 16-node cluster but need to mount only the file system from two nodes, you need only two journals. If you need to mount from a third node, you can always add a journal with the gfs2_jadd command. With GFS2, you can add journals on the fly.

See https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/global_file_system_2/s1-manage-addjournalfs

Mount fails when adding new node to GFS2 EBS multi-attached Clustered storage

Relevant content