Java Wget Bz2 file

Question

I'm trying to webget some bz2 files from Wikipedia, I don't care whether they are save as bz2 or unpacked, since I can unzip them locally.

When I call:

public static void getZip(String theUrl, String filename) throws IOException {
    URL gotoUrl = new URL(theUrl);
    try (InputStreamReader isr = new InputStreamReader(new BZip2CompressorInputStream(gotoUrl.openStream())); BufferedReader in = new BufferedReader(isr)) {
        StringBuffer sb = new StringBuffer();
        String inputLine;

        // grab the contents at the URL
        while ((inputLine = in.readLine()) != null) {
            sb.append(inputLine + "\r\n");
        }
        // write it locally
        Wget.createAFile(filename, sb.toString());
    } catch (MalformedURLException mue) {
        mue.printStackTrace();
    } catch (IOException ioe) {
        throw ioe;
    }
}

I get a part of the unzipped file, never more than +- 883K.
When I don't use the BZip2CompressorInputStream, like:

public static void get(String theUrl, String filename) throws IOException {
    try {
        URL gotoUrl = new URL(theUrl);
        InputStreamReader isr = new InputStreamReader(gotoUrl.openStream());
        BufferedReader in = new BufferedReader(isr);

        StringBuffer sb = new StringBuffer();
        String inputLine;

        // grab the contents at the URL
        while ((inputLine = in.readLine()) != null) {
            sb.append(inputLine);// + "\r\n");
        }
        // write it locally
        Statics.writeOut(filename, false, sb.toString());
    } catch (MalformedURLException mue) {
        mue.printStackTrace();
    } catch (IOException ioe) {
        throw ioe;
    }
}

I get a file of which the size is the same as it suppose to (compared to the KB not B). But also a message that that the zipped file is damaged, also when using byte [] instead of readLine(), like:

public static void getBytes(String theUrl, String filename) throws IOException {
    try {
        char [] cc = new char[1024];
        URL gotoUrl = new URL(theUrl);
        InputStreamReader isr = new InputStreamReader(gotoUrl.openStream());
        BufferedReader in = new BufferedReader(isr);

        StringBuffer sb = new StringBuffer();
        // grab the contents at the URL
        int n = 0;
        while (-1 != (n = in.read(cc))) {
            sb.append(cc);// + "\r\n");
        }
        // write it locally
        Statics.writeOut(filename, false, sb.toString());
    } catch (MalformedURLException mue) {
        mue.printStackTrace();
    } catch (IOException ioe) {
        throw ioe;
    }
}

Finally, when I bzip2 the inputstream and outputstream, I get a valid bzip2 file, but of the size like the first one, using:

public static void getWriteForBZ2File(String urlIn, final String filename) throws CompressorException, IOException {
    URL gotoUrl = new URL(urlIn);
    try (final FileOutputStream out = new FileOutputStream(filename);
            final BZip2CompressorOutputStream dataOutputStream = new BZip2CompressorOutputStream(out);
            final BufferedInputStream bis = new BufferedInputStream(gotoUrl.openStream());
            final CompressorInputStream input = new CompressorStreamFactory().createCompressorInputStream(bis);
            final BufferedReader br2 = new BufferedReader(new InputStreamReader(input))) {
        String line = null;
        while ((line = br2.readLine()) != null) {
            dataOutputStream.write(line.getBytes());
        }
    }
}

So, how do I get the entire bz2 file, in either bz2 format or unzipped?

Answer 1

A bz2 file contains bytes, not characters. You can't read it as if it contained characters, with a Reader.

Since all you want to do is download the file and save it locally, all you need is

Files.copy(gotoUrl.openStream(), Paths.get(fileName));