Efficiently read file from URL into byte[] in Java

Question

I'm trying to find a more efficient method of reading a file from a remote URL and saving it into a byte array. Here is what I currently have:

private byte[] fetchRemoteFile(String location) throws Exception {
  URL url = new URL(location);
  InputStream is = null;
  byte[] bytes = null;
  try {
    is = url.openStream ();
    bytes = IOUtils.toByteArray(is);
  } catch (IOException e) {
    //handle errors
  }
  finally {
    if (is != null) is.close();
  }
  return bytes;
}

As you can see, I currently pass the URL into the method, where it uses an InputStream object to read in the bytes of the file. This method uses Apache Commons IOUtils. However, this method call tends to take a relatively long time to run. When retrieving hundreds, thousands, or hundreds of thousands of files one right after another, it gets quite slow. Is there a way I could improve this method so that it runs more efficiently? I have considered multithreading but I would like to save that as a last resort.

Answer 1

Your way of doing it seems like absolutely ok.

But if you saying:

"However, this method call tends to take a relatively long time to run"

You can have follow problems :

• Network, connection issue

• Are you sure that download each file in separate thread?

If you are using multithreading for that, be sure that VM args -XmsYYYYM and -XmxYYYYM configured well, because if not you can face problem , that your processor not using all cores. I have faced this problem some time ago.