File upload/download using nfs-client package not working after n iteration

Question

I am using nfs-client package to realize the NFS protocol to connect to the linux server for file upload/download. Basically the purpose is NFS protocol testing .

This is my sample codebase

import com.emc.ecs.nfsclient.nfs.io.Nfs3File;
import com.emc.ecs.nfsclient.nfs.io.NfsFileInputStream;
import com.emc.ecs.nfsclient.nfs.io.NfsFileOutputStream;
import com.emc.ecs.nfsclient.nfs.nfs3.Nfs3;
import com.emc.ecs.nfsclient.rpc.CredentialUnix;
int thread_count = ctx.getThreadNum();
String lrandom_name = service_name + "_" + replica_id + "_" + thread_count;

public static void uploadFileToNfs() {
        String localDir = "F:\\look\\"+lrandom_name;
        InputStream inputStream = null;
        OutputStream outputStream = null;
        try {
            //Create a local file object
            File localFile = new File(localDir);
            //Get the file name of the local file, this name is used to create a file with the same name in the specified directory on the remote Nfs server
            String localFileName = localFile.getName();
            Nfs3 nfs3 = new Nfs3(NFS_IP, NFS_DIR, new CredentialUnix(0, 0, null), 3);
            //Create Nfs file object on remote server
            Nfs3File NfsFile = new Nfs3File(nfs3, "/" + localFileName);
            //Open a file input stream
            inputStream = new BufferedInputStream(new FileInputStream(localFile));
            //Open a remote Nfs file output stream and copy the file to the destination
            outputStream = new BufferedOutputStream(new NfsFileOutputStream(NfsFile));

            //Buffer memory
            byte[] buffer = new byte[1024];
            while ((inputStream.read(buffer)) != -1) {
                outputStream.write(buffer);
            }
            System.out.println("File upload complete!");
        } catch (Exception ex) {
            ex.printStackTrace();
        } finally {
            try {
                if (outputStream != null) {
                    outputStream.close();
                }
                if (inputStream != null) {
                    inputStream.close();
                }
            } catch (IOException ex) {
                ex.printStackTrace();
            }
        }
    }

I am running the script from Jmeter , but after n number of iteration (threaded manner) I am getting below error. Similar script for File download from NFS is working fine in threaded manner parallelly .

com.emc.ecs.nfsclient.nfs.NfsException: rpc error, server: 192.168.0.101, RPC error: RPC call is ACCEPTED, but the status is not success, acceptStat=4
        at com.emc.ecs.nfsclient.rpc.RpcWrapper.handleRpcException(RpcWrapper.java:311)
        at com.emc.ecs.nfsclient.rpc.RpcWrapper.callRpcWrapped(RpcWrapper.java:159)
        at com.emc.ecs.nfsclient.nfs.nfs3.Nfs3.wrapped_sendWrite(Nfs3.java:756)
        at com.emc.ecs.nfsclient.nfs.nfs3.Nfs3.wrapped_sendWrite(Nfs3.java:90)
        at com.emc.ecs.nfsclient.nfs.io.NfsFileBase.write(NfsFileBase.java:866)
        at com.emc.ecs.nfsclient.nfs.io.NfsFileOutputStream.writeBufferToFile(NfsFileOutputStream.java:296)
        at com.emc.ecs.nfsclient.nfs.io.NfsFileOutputStream.write(NfsFileOutputStream.java:262)
        at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81)
        at java.base/java.io.BufferedOutputStream.write(BufferedOutputStream.java:127)
        at java.base/java.io.FilterOutputStream.write(FilterOutputStream.java:108)
        at java_io_FilterOutputStream$write.call(Unknown Source)
        at com.emc.ecs.nfsclient.nfs.io.Script1.run(Script1.groovy:77)
        at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:317)
        at org.codehaus.groovy.jsr223.GroovyCompiledScript.eval(GroovyCompiledScript.java:71)
        at java.scripting/javax.script.CompiledScript.eval(CompiledScript.java:89)
        at org.apache.jmeter.util.JSR223TestElement.processFileOrScript(JSR223TestElement.java:217)
        at org.apache.jmeter.protocol.java.sampler.JSR223Sampler.sample(JSR223Sampler.java:72)
        at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:635)
        at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:558)
        at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:489)
        at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256)
        at java.base/java.lang.Thread.run(Thread.java:834)

Update Based on feedback instead of one static file for all Threads(Users) , now I have unique file for each Threads(Users) to upload/download

Update

Tried below one more use case randomly picking file from one NFS share folder to other ,same issue coming. Suspecting something wrong with NFS configuration only

package com.emc.ecs.nfsclient.nfs.io;

import org.junit.Test;
import com.emc.ecs.nfsclient.nfs.NfsSetAttributes;
import com.emc.ecs.nfsclient.nfs.io.Nfs3File;
import com.emc.ecs.nfsclient.nfs.io.NfsFileInputStream;
import com.emc.ecs.nfsclient.nfs.io.NfsFileOutputStream;
import com.emc.ecs.nfsclient.nfs.nfs3.Nfs3;
import com.emc.ecs.nfsclient.rpc.CredentialUnix;
import java.util.Random;


import java.io.IOException;

import org.junit.Test;
String nfs_folder = vars.get("nfs_folder");
String files_folder = vars.get("files_folder");
String NFS_IP = vars.get("server_ip");
String NFS_DIR = "/"+nfs_folder+"/"+files_folder;
String NFS_DIR_ROOT = "/"+nfs_folder;

int thread_count = ctx.getThreadNum();
String service_name = vars.get("Service_Name");
String replica_id = vars.get("Replica_ID");
String lrandom_name = service_name + "_" + replica_id + "_" + thread_count;
int transfer_size = Integer.parseInt(vars.get("transfer_size"))


InputStream inputStream = null;
OutputStream outputStream = null;
try {
  Nfs3 nfs3 = new Nfs3(NFS_IP, NFS_DIR, new CredentialUnix(0, 0, null), 3);
  Nfs3 nfs4 = new Nfs3(NFS_IP, NFS_DIR_ROOT, new CredentialUnix(0, 0, null), 3);
  
  nfsFiles = new Nfs3File(nfs3,"/");
  List<String> children = nfsFiles.list();
  Random randomGenerator = new Random();
  int fileIndex = randomGenerator.nextInt(children.size());
   OUT.println(fileIndex)
  String NfsFileDir = "/"+children.get(fileIndex);
  OUT.println(children.size())
  OUT.println(children.get(fileIndex))
  //Create Nfs file object on remote server
  Nfs3File nfsFile = new Nfs3File(nfs3, NfsFileDir);
  Nfs3File NfsFile = new Nfs3File(nfs4, "/" + lrandom_name);
  //String localFileName = localDir + lrandom_name;
  //Create a local file object
  //File localFile = new File(localFileName);
  //Open a file input stream
  inputStream = new BufferedInputStream(new NfsFileInputStream(nfsFile));
  //Open a remote Nfs file output stream and copy the file to the destination
  outputStream = new BufferedOutputStream(new NfsFileOutputStream(NfsFile));

  //Buffer memory
  byte[] buffer = new byte[transfer_size];

  while (inputStream.read(buffer) != -1) {
      outputStream.write(buffer);
  }
  System.out.println("File copy complete!");
} catch (IOException ex) {
  ex.printStackTrace();
} finally {
  try {
      if (outputStream != null) {
          outputStream.close();
      }
      if (inputStream != null) {
          inputStream.close();
      }
  } catch (Exception e) {
      e.printStackTrace();
  }
}

Answer 1

1. I don't think your use case is valid as uploading the same file into the same destination is not something what real NFS users would be doing, so maybe it worth considering using i.e. Directory Listing Config plugin so each virtual user will have its own file(s) to upload/download

2. Looking at NFS RPC RFC the status you're getting is:

   GARBAGE_ARGS  = 4, /* procedure can't decode params         */
   

   so double check that your JMeter instance has enough headroom to operate in terms of JVM setup, CPU, RAM, etc. as it looks like that your NFS server cannot properly parse the request so it might be the case JMeter sends some trash. Ensure to follow JMeter Best Practices

3. Basically the same as point 2 but for NFS server side, it might be the case it is overloaded hence cannot properly parse the incoming valid requests, check NFS logs and your operating system logs for any suspicious entries.