Add up numbers in files

Question

I have a directory with a lot of files (~40,000), and each file has exactly two lines, each with a number. I want to add up all of the numbers in the entire directory; how do I do that fast and efficiently?

I tried this, but it doesn't work, and I can't figure out for the life of me why. I get a NullPointerException, but it shouldn't be, since I'm guessing that the listOfFiles.length is causing it.

package me.counter;

import java.io.File;
import java.io.IOException;
import java.io.PrintWriter;
import java.util.Scanner;

public class TicTacToeCounter {
    static String dir = "./data/";
    public static void main(String args[]) throws IOException{
        int total = 0;
        File folder = new File(dir);
        File[] listOfFiles = folder.listFiles();

            for (int i = 0; i < listOfFiles.length; i++) {
                total += getWins(listOfFiles[i].getAbsolutePath());
                total += getLosses(listOfFiles[i].getAbsolutePath());
            }

            System.out.println(total);
    }

    public static int getWins(String move) throws IOException{
        File f = new File(move);
        if(!f.exists()){
            f.createNewFile();
            PrintWriter writer = new PrintWriter(move, "UTF-8");
            writer.println("0");
            writer.println("0");
            writer.close();
            return 0; 
        }
        Scanner fscanner = new Scanner(f);
        int wins = 0;
        if(fscanner.hasNext())
            wins = fscanner.nextInt();
        fscanner.close();
        return wins;
    }

    public static int getLosses(String move) throws IOException{
        File f = new File(move);
        if(!f.exists()){
            f.createNewFile();
            PrintWriter writer = new PrintWriter(move, "UTF-8");
            writer.println("0");
            writer.println("0");
            writer.close();
            return 0; 
        }
        Scanner fscanner = new Scanner(f);
        fscanner.nextInt();
        int losses = 0; 
        if(fscanner.hasNext())
            losses = fscanner.nextInt();
        fscanner.close();
        return losses;
    }
}

Answer 1

This is exactly what you need. It will check all files dynamically, you don't need to give number of files. for example, if you have any number of files and various number of rows in each file, not to worry. it will read it correctly.

import java.io.*;
import java.util.*;
import java.text.*;

public class TicTacToeCounter
{
    //arraylist that will read and hold all file names that can be used later in the program.
    public ArrayList<String> fileList = new ArrayList<String>();
    //arraylist of all lines from all files.
    ArrayList<Integer> theData = new ArrayList<Integer>();

    //class constructor
    public TicTacToeCounter() throws Exception
    {
        //provide the directory name here where you have those 40000 files.
        //I have testdata directory in my program in the same folder where my .java and .class file resides.
        File folder = new File("testdata");
        File[] listOfFiles = folder.listFiles();
        ArrayList<String> tempData = new ArrayList<String>();

        for (int i = 0; i < listOfFiles.length; i++)
        {
            if (listOfFiles[i].isFile())
            {
                fileList.add(listOfFiles[i].getName());
            }
            else if (listOfFiles[i].isDirectory())
            {
                System.out.println("Directory " + listOfFiles[i].getName());
            }
        }

        //for every filename in fileList, do....
        for (String s : fileList)
        {
            //call readFile method and pass file name as a variable s. add objects to tempData arraylist.
            tempData = readFile(s);
            //for every line in tempData, do....
            for (String line : tempData)
            {
                //add the line in theData arraylist, also convert into Integer before adding.
                theData.add(Integer.parseInt(line));
            }
        }

        //for every object in theData arraylist, print the object. alternatevely you can print it in previous stage.
        for (Integer s : theData)
        {
            System.out.println(s);
        }
    }

    //readFile method that will read our data files.
    public ArrayList<String> readFile(String fileName) throws Exception
    {
        ArrayList<String> data = new ArrayList<String>();
        //don't forget to add directory name here as we are only passing filename, not directory.
        BufferedReader in = new BufferedReader(new FileReader("testdata/"+fileName));

        String temp = in.readLine(); 
        while (temp != null)
        {
            data.add(temp);
            temp = in.readLine(); 
        }
        in.close();
        return data;
    }
}

Answer 2

Stack trace should tell you the line number exactly where the error happens and you don't have to guess. Please check: the directory exists and it's a directory and your listOfFiles is not null before you do a length on it.

folder.exists() && folder.isDirectory() {
   \\ you might want to check if folder.canRead() and folder.canWrite()
   \\ get listOfFiles
}

if (listOfFiles != null) { // proceed with operations

P.S: Also your getWins and getLosses could be improved. I'd probably try reading the file once (and create if those doesn't exist, if you must but as @sstan mentioned you just got the file name from the directory, there is no reason why it shouldn't exist) and read both wins and losses if there are always only 2 lines in the files. Right now, you are creating one if it doesn't exist and reading the one you just created right after, which we don't have to.

Answer 3

Instead of using array for File, use ArrayList. It is more convenient and transparent to give you an idea why the error occurs. and also, if you use arrays, you must define a length when you initiate it, which you hasn't been done in your case.