"Read-only file system" error while creating snapshot in elasticsearch with multiple-nodes

Question

I have setup an Elasticsearch cluster with 2 nodes. I'm trying to implement the Snapshot-Restore process now. But I get this "Read-only file system" Error. And I'm unable to create the snapshot because of this error.

I have created a directory in 1st node with 777 permission. Used NFS and shared the directory with the 2nd node. I am able to navigate to that shared directory in the 2nd node terminal. I have also set path.repo on both nodes to point to that shared directory[/opt/backups]. But still getting the "Read-only". I cannot understand why.

I make a request to create snapshot:

PUT host.com/elsearch/_snapshot/backup_1
{
    "type": "fs",
    "settings": {
        "location": "/opt/backup",
        "compress": true
    }
}

This is the error :

{
    "error": {
        "root_cause": [
            {
                "type": "repository_verification_exception",
                "reason": "[backup_1] [[jhsdgfjgeufh, 'RemoteTransportException[[es-node-2][26.19.35.46:9300][internal:admin/repository/verify]]; nested: ElasticsearchException[failed to create blob container]; nested: FileSystemException[/opt/backups/tests-Dvl-pSO2Cg: Read-only file system];']]"
            }
        ],
        "type": "repository_verification_exception",
        "reason": "[backup_1] [[jhsdgfjgeufh, 'RemoteTransportException[[es-node-2][26.19.35.46:9300][internal:admin/repository/verify]]; nested: ElasticsearchException[failed to create blob container]; nested: FileSystemException[/opt/backups/tests-Dvl-pSO2Cg: Read-only file system];']]"
    },
    "status": 500
}

tests-Dvl-pSO2Cg directory has got created on the main server.

I have also tried, since I saw this solution in some of the similar questions :

chown -R elasticsearch:elasticsearch $BACKUP_DIR

Answer 1

I was able to solve the problem by executing these commands :

setfacl -R -d -m u::rwx $BACKUP_DIR
setfacl -R -d -m g::rwx $BACKUP_DIR
setfacl -R -d -m o::rwx $BACKUP_DIR

Basically, the directories that were getting created (when I call the PUT '/_snapshot/...') inside my $BACKUP_DIR were not taking the 777 permission, though I had given chmod 777 with -R. The commands above will handle the issue.