How to write over 2000,000 number of files in SDCARD on Linux

Question

How to write over 2000,000 number of files in SDCARD on Linux.

I am developing a program on Fedora 9. This program write over 2000,000 number of files to SDCARD. Here is this program code.

int vnCount = 0;
int vnDirCount = 0;
int vnSubDirCount = 0;
int vnTotalCounts = 3600;
int vnTotalSubDirs = 8;
int vnTotalDirs = 1;

int vnWriteNum;
char vstrTmp[256];
memset(&vstrTmp[0], 0x00, sizeof(vstrTmp));

FILE *pfd;
while (vnDirCount < vnTotalDirs) {

    memset(&vDstDirName[0], 0x00, sizeof(vDstDirName));

    sprintf( &vDstDirName[0], "/media/SD/log/%08d",  vnDirCount);
    printf("<<< output dir: %s\n", &vDstDirName[0]);
    if( mkdir( vDstDirName, 0x644 ) < 0 ) {

        printf(">>> output dir: error 0 : %s\n", vDstDirName);
        return -1;
    }

    usleep(1000*10);

    vnSubDirCount = 0;
    while(vnSubDirCount < vnTotalSubDirs) {

        memset(&vSubDirName[0], 0x00, sizeof(vSubDirName)); 

        sprintf(&vSubDirName[0], "%s/%02d", &vDstDirName[0], vnSubDirCount);
        printf("??? output dir: %s\n", &vSubDirName[0]);

        if( mkdir( vSubDirName, 0x777 ) < 0 ) {
            printf("/// output dir:  error : %s\n", vSubDirName);
            return -1;
        }

        vnCount = 0;
        while(vnCount < vnTotalCounts) {
            memset(&vstrTmp[0], 0x00, sizeof(vstrTmp));
            sprintf(&vstrTmp[0], "%s/%08d%06d.dat", &vSubDirName[0], vnCount);

            printf("*** output file: %s\n", &vstrTmp[0]);

            if ((pfd = fopen(&vstrTmp[0], "w")) == NULL)
            {
                printf(" @@@ Cannot open output file \n");
                return 1;
            }

            memset(&buf[0], 0xaa, sizeof(buf));
            printf("output file:1 \n");

            vnWriteNum = fwrite(&buf[0], sizeof(buf), 1, pfd);

            //ret = write(pfd, &buf[0], sizeof(buf));

            printf("output file:2 \n");
            if (vnWriteNum != 1) {
                fclose(pfd);
                printf(" $$$ Cannot write output file \n");
                return 1;
            }


            if (fflush(pfd) != 0) {
              printf(" ^^^ Cannot flush output file \n");
            }

            if (fsync(fileno(pfd)) != 0) {
              printf(" ^^^ Cannot flush output file \n");
            }


            printf("output file:3 \n");

            if (fclose(pfd) != 0) {
              printf(" ### Cannot close output file \n");
            }

            usleep(1000*1); 

            vnCount = vnCount +1;
        }

        vnSubDirCount = vnSubDirCount + 1;
    }

    vnDirCount = vnDirCount + 1;
}

The program is write successful until 2000000 number of files. But I can't see over 23000 number of files in SDCARD on Linux konqueror. Only can see 23000 number of files. At konsole is too same. So I do check the SDCARD by "fsck.vfat" linux command. Here is the command result.

fsck.vfat /dev/sdd1
dosfsck 2.11, 12 Mar 2005, FAT32, LFN
Orphaned long file name part "00000000000746.dat"
1: Delete.
2: Leave it.
? 1
Reclaimed 7658 unused clusters (31367168 bytes).
Free cluster summary wrong (479444 vs. really 487102)
1) Correct
2) Don't correct

How to write over 23000 number of file on SDCARD.

Each file's size is 3.5K. SDCARD's capacity is 2G. This SDCARD was format to FAT32(allocation unit 4K) on Windows. Linux system is Fedora 9. Can anyone help me. Thanks.

Answer 1

Don't do that, the FAT filesystem is not good for huge directories and many files (so I believe you cannot do what you want). And SDCARD flash hardware dislikes doing a big lot of writes!

You could consider other approaches, e.g. using Sqlite or GDBM; then at least you'll write one single file.

Sqlite can be used on any filesystem, even as primitive as a FAT one. It only uses one file (and another one for journal). Sqlite is essentially a library giving you SQL abilities on a database stored in one single file. And likewise for GDBM (it uses one or two files). Of course the filesystem might limit (4Gb for FAT32) the maximal size of a file. Read some sqlite tutorial, the sqlite documentations, the GDBM manual.

You could also use some different filesystem on your SDCARD (e.g. ext3 or ext4 or btrfs, which you'll read on Linux mostly).

See also this & that answers.

Answer 2

You could also just use another file system. Using fdisk or gparted you should be able to manipulate the partisons. Maybe ext4 or something else has higher limits.

But in general, the other answer is right, you should avoid doing that and look for a better solution. It might get slow and unhandy using that many files. Can't you store all in one file?

Answer 3

The fat32 filesystem has several limitations, and one is about the Max number of files : 268,173,300 (2^28ish)

As explained by basile, using a database such as sqlite is a possibility, but keep in mind that a file mustn't exceed 4GB on fat32 (or ~260 on fat32+)

You could also be interested in having a virtual filesystem : Simple Virtual Filesystem in C/C++