Encoding cp-1252 as utf-8?

Question

I am trying to write a Java app that will run on a linux server but that will process files generated on legacy Windows machines using cp-1252 as the character set. Is there anyway to encode these files as utf-8 instead of the cp-1252 it is generated as?

Answer 1

You can read and write text data in any encoding that you wish. Here's a quick code example:

  public static void main(String[] args) throws Exception
  {
    // List all supported encodings
    for (String cs : Charset.availableCharsets().keySet())
      System.out.println(cs);

    File file = new File("SomeWindowsFile.txt");
    StringBuilder builder = new StringBuilder();

    // Construct a reader for a specific encoding
    Reader reader = new InputStreamReader(new FileInputStream(file), "windows-1252");
    while (reader.ready())
    {
      builder.append(reader.read());
    }
    reader.close();

    String string = builder.toString();

    // Construct a writer for a specific encoding
    Writer writer = new OutputStreamWriter(new FileOutputStream(file), "UTF8");
    writer.write(string);
    writer.flush();
    writer.close();
  }

If this still 'chokes' on read, see if you can verify that the the original encoding is what you think it is. In this case I've specified windows-1252, which is the java string for cp-1252.

Answer 2

If the file names as well as content is a problem, the easiest way to solve the problem is setting the locale on the Linux machine to something based on ISO-8859-1 rather than UTF-8. You can use locale -a to list available locales. For example if you have en_US.iso88591 you could use:

export LANG=en_US.iso88591

This way Java will use ISO-8859-1 for file names, which is probably good enough. To run the Java program you still have to set the file.encoding system property:

java -Dfile.encoding=cp1252 -cp foo.jar:bar.jar blablabla

If no ISO-8859-1 locale is available you can generate one with localedef. Installing it requires root access though. In fact, you could generate a locale that uses CP-1252, if it is available on your system. For example:

sudo localedef -f CP1252 -i en_US en_US.cp1252
export LANG=en_US.cp1252

This way Java should use CP1252 by default for all I/O, including file names.

Expanded further here: http://jonisalonen.com/2012/java-and-file-names-with-invalid-characters/