BackupAgentHelper crashing with large numbers of files

Question

I have an app that has large quantities of files generated by the user that I need to backup. All content is stored in the db, preferences or 1 of 4 folders. I wrote a simple helper function to generate all the FileBackupHelpers for the files in each folder. However, it seems that after my onCreate() has run the framework code for FileBackupHelper is choking on the quantity of paths to parse through and crashing my app.

Here is the code for my backup agent:

package <packageNameRemovedToProtectClient>.service;

import android.app.backup.BackupAgentHelper;
import android.app.backup.FileBackupHelper;
import android.app.backup.SharedPreferencesBackupHelper;
import android.content.Context;
import android.os.Environment;
import android.util.Log;

import <packageNameRemovedToProtectClient>.data.DBManager;

import java.io.File;
import java.util.Arrays;
import java.util.Collections;
import java.util.Vector;

/**
 * Created by mclark4386 on 9/18/14.
 */
public class AABackupAgent extends BackupAgentHelper {
private static final String CLASS_NAME = AABackupAgent.class.getSimpleName();

private Context mContext;

public AABackupAgent() {
    super();
}

@Override
public void onCreate() {
    super.onCreate();
    mContext = this;
    AssetManager.setupSingleton(mContext);
    Log.v(CLASS_NAME,"GO!");
    this.addHelper("prefs", new SharedPreferencesBackupHelper(mContext, Service.PREFS_NAME));
    String dbfile = DBManager.filePath.replace(".sqlite", "");
    Log.v(CLASS_NAME,"DB file:"+dbfile);
    File dataDirectory = Environment.getDataDirectory();
    File db = new File(dataDirectory,DBManager.fullDBPath);
    Log.v(CLASS_NAME,"path:"+db.getAbsolutePath());
    this.addHelper("db", new FileBackupHelper(mContext, db.getAbsolutePath()));
    AddFolderHelper(AssetManager.thingImageFolder.getAbsolutePath());
    AddFolderHelper(AssetManager.userFolder.getAbsolutePath());
    AddFolderHelper(AssetManager.categoryImageFolder.getAbsolutePath());
    AddFolderHelper(AssetManager.audioFolder.getAbsolutePath());
    Log.v(CLASS_NAME,"onCreate Done!");
}

private void AddFolderHelper(String path){
    try{
        File folder = new File(path);
        if (folder.isDirectory()){
            Log.v(CLASS_NAME,"Filebackup list size::"+folder.list().length+" for path:"+folder.getName());
            File[] listFiles = folder.listFiles();
            Vector<String> filepaths = new Vector<String>(listFiles.length);
            for (int i = 0;i< listFiles.length;i++)
                filepaths.add(listFiles[i].getAbsolutePath());
            if (filepaths.size()>512){
                int count = (filepaths.size()/511);
                if(filepaths.size()%511 > 0)count++;
                Log.v(CLASS_NAME,"should break up the paths in to "+count+" chunks!");
                String[] paths = new String[filepaths.size()];
                filepaths.toArray(paths);
                String shortName = folder.getName();
                for (int i = 0; i < count; i++) {
                    Log.v(CLASS_NAME,"adding helper for path:"+shortName+i);
                    String[] pathRange = Arrays.copyOfRange(paths, 511 * i, (511 * i) + 511);
                    this.addHelper(shortName+i, new FileBackupHelper(mContext, pathRange));
                }
            }else {
                String[] paths = new String[filepaths.size()];
                filepaths.toArray(paths);
                this.addHelper(path, new FileBackupHelper(mContext, paths));
            }
        }else{
            this.addHelper(path, new FileBackupHelper(mContext, path));
        }
    }catch (Exception e){
        Log.e(CLASS_NAME,"ERROR: AddFolderHelper:");
        e.printStackTrace();
    }
}
}

Here is the Log output:

V/AABackupAgent﹕ GO!
V/AABackupAgent﹕ DB file:AADataModel
V/AABackupAgent﹕ path:/data/data/<packageNameRemovedToProtectClient>/databases/AADataModel.sqlite
V/AABackupAgent﹕ Filebackup list size::551 for path:thingImage
V/AABackupAgent﹕ should break up the paths in to 2 chunks!
V/AABackupAgent﹕ adding helper for path:thingImage0
V/AABackupAgent﹕ adding helper for path:thingImage1
V/AABackupAgent﹕ Filebackup list size::1 for path:user
V/AABackupAgent﹕ Filebackup list size::0 for path:categoryImage
V/AABackupAgent﹕ Filebackup list size::551 for path:audio
V/AABackupAgent﹕ should break up the paths in to 2 chunks!
V/AABackupAgent﹕ adding helper for path:audio0
V/AABackupAgent﹕ adding helper for path:audio1
V/AABackupAgent﹕ onCreate Done!
V/BackupServiceBinder﹕ doBackup() invoked
D/BackupHelperDispatcher﹕ handling existing helper 'db' android.app.backup.FileBackupHelper@4336b8e8
D/BackupHelperDispatcher﹕ handling existing helper 'prefs' android.app.backup.SharedPreferencesBackupHelper@43595ce8
D/BackupHelperDispatcher﹕ handling new helper '/storage/emulated/0/Android/data/<packageNameRemovedToProtectClient>/files/categoryImage'
D/BackupHelperDispatcher﹕ handling new helper '/storage/emulated/0/Android/data/<packageNameRemovedToProtectClient>/files/user'
D/BackupHelperDispatcher﹕ handling new helper 'audio0'
E/dalvikvm﹕ JNI ERROR (app bug): local reference table overflow (max=512)
W/dalvikvm﹕ JNI local reference table (0x7e2ff770) dump:
W/dalvikvm﹕ Last 10 entries (of 512):
W/dalvikvm﹕ 511: 0x426c0878 java.lang.String "/data/data/com.m... (132 chars)
W/dalvikvm﹕ 510: 0x426c0830 java.lang.String "/data/data/com.m... (138 chars)
W/dalvikvm﹕ 509: 0x426c0500 java.lang.String "/data/data/com.m... (127 chars)
W/dalvikvm﹕ 508: 0x426c01d8 java.lang.String "/data/data/com.m... (131 chars)
W/dalvikvm﹕ 507: 0x426ba7d8 java.lang.String "/data/data/com.m... (131 chars)
W/dalvikvm﹕ 506: 0x426ba790 java.lang.String "/data/data/com.m... (137 chars)
W/dalvikvm﹕ 505: 0x426bf488 java.lang.String "/data/data/com.m... (135 chars)
W/dalvikvm﹕ 504: 0x426bf158 java.lang.String "/data/data/com.m... (130 chars)
W/dalvikvm﹕ 503: 0x426bee38 java.lang.String "/data/data/com.m... (129 chars)
W/dalvikvm﹕ 502: 0x426b94b0 java.lang.String "/data/data/com.m... (131 chars)
W/dalvikvm﹕ Summary:
W/dalvikvm﹕ 1 of java.lang.Class
W/dalvikvm﹕ 507 of java.lang.String (507 unique instances)
W/dalvikvm﹕ 2 of java.io.FileDescriptor (2 unique instances)
W/dalvikvm﹕ 2 of java.lang.String[] (511 elements) (2 unique instances)
E/dalvikvm﹕ Failed adding to JNI local ref table (has 512 entries)
I/dalvikvm﹕ "Binder_3" prio=5 tid=20 RUNNABLE
I/dalvikvm﹕ | group="main" sCount=0 dsCount=0 obj=0x42f4b218 self=0x777b2a98
I/dalvikvm﹕ | sysTid=10236 nice=0 sched=0/0 cgrp=apps handle=1942542136
I/dalvikvm﹕ | state=R schedstat=( 18035885 5737305 43 ) utm=1 stm=0 core=3
I/dalvikvm﹕ at android.app.backup.FileBackupHelperBase.performBackup_native(Native Method)
I/dalvikvm﹕ at android.app.backup.FileBackupHelperBase.performBackup_checked(FileBackupHelperBase.java:76)
I/dalvikvm﹕ at android.app.backup.FileBackupHelper.performBackup(FileBackupHelper.java:85)
I/dalvikvm﹕ at android.app.backup.BackupHelperDispatcher.doOneBackup(BackupHelperDispatcher.java:95)
I/dalvikvm﹕ at android.app.backup.BackupHelperDispatcher.performBackup(BackupHelperDispatcher.java:76)
I/dalvikvm﹕ at android.app.backup.BackupAgentHelper.onBackup(BackupAgentHelper.java:66)
I/dalvikvm﹕ at android.app.backup.BackupAgent$BackupServiceBinder.doBackup(BackupAgent.java:567)
I/dalvikvm﹕ at android.app.IBackupAgent$Stub.onTransact(IBackupAgent.java:80)
I/dalvikvm﹕ at android.os.Binder.execTransact(Binder.java:404)
I/dalvikvm﹕ at dalvik.system.NativeStart.run(Native Method)
I/dalvikvm﹕ [ 09-20 04:06:45.777  9815:10236 E/dalvikvm ] VM aborting
A/libc﹕ Fatal signal 6 (SIGABRT) at 0x00002657 (code=-6), thread 10236 (Binder_3)
W/ContextImpl﹕ Failed to ensure directory: /storage/extSdCard/Android/data/<packageNameRemovedToProtectClient>/files
W/ContextImpl﹕ Failed to ensure directory: /storage/extSdCard/Android/data/<packageNameRemovedToProtectClient>/files
V/AABackupAgent﹕ GO!
V/AABackupAgent﹕ DB file:AADataModel
V/AABackupAgent﹕ path:/data/data/<packageNameRemovedToProtectClient>/databases/AADataModel.sqlite
V/AABackupAgent﹕ Filebackup list size::551 for path:thingImage
V/AABackupAgent﹕ should break up the paths in to 2 chunks!
V/AABackupAgent﹕ adding helper for path:thingImage0
V/AABackupAgent﹕ adding helper for path:thingImage1
V/AABackupAgent﹕ Filebackup list size::1 for path:user
V/AABackupAgent﹕ Filebackup list size::0 for path:categoryImage
V/AABackupAgent﹕ Filebackup list size::551 for path:audio
V/AABackupAgent﹕ should break up the paths in to 2 chunks!
V/AABackupAgent﹕ adding helper for path:audio0
V/AABackupAgent﹕ adding helper for path:audio1
V/AABackupAgent﹕ onCreate Done!

Thanks in advance for any help you can give! If you need any other information please let me know.

Answer 1

Here are the details of what I did. At the beginning I tried the same as you, I implemented BackupAgentHelper but I realized quickly it won't be enough to backup and restore files, I think it is just ok for really light data or the user preferences. In my case I needed to backup a SQLite database and many pictures related.

Here is basically the process I setup:

For backups:

1. First step: one AsyncTask copy and gather the different files to backup in the same folder on the external storage.
2. Second step: another AsyncTask zip all the files together
3. Third step: another AsyncTask transfer the zip to Dropbox (I think I might also do it with Google Drive later)

For restorations:

1. First step: one AsyncTask transfer the zip from Dropbox to the device.
2. Second step: another AsyncTask unzip all the files
3. Third step: another AsyncTask moves the files where they should be.

This way I am sure all is made on time, and I can follow what happens. With BackupAgentHelper, you cannot follow easily when and how it is done.

PS: For Dropbox, I used the Core API, they give a good project example ("DBRoulette") with the libraries provided.