C# - ESENT db memory leaks?

Question

I just started to use ESENT ManagedInterface (http://managedesent.codeplex.com/). I am wondering whether it has a memory leak issue.

What I am doing is fairly simple. I followed sample code, but I put a quite large string data (10KB+) in every row and produce 10000 rows in total.

The memory usage increases while more rows are inserted. If I insert like 100,000 rows, the program will eat up 1 GB memory and dies.

Here is the code.

public static void test()
    {
        string techcrunchString = @"The Latest from TechCrunch CMU Researchers Turn Any Surface Into A TouchscreenWeb Design Community Treehouse Raises $600K From Reid Hoffman, Kevin Rose, And Others Greylock Looks To Help Portfolio Companies Recruit Talent With New Hires UberMedia Quietly (Inadvertently?) Releases Chime.in, A Mobile Social Networking App T-Mobile Announces The Dual-Screen LG DoublePlay, Launching November 2nd? Watch An iPhone 4S and Samsung Galaxy S II Take Three Nasty Drops Onto Concrete Facebook, NRDC & Opower To Partner On Energy-Saving Social AppCTIAs New Alert Guidelines Could Mean The End Of Bill ShockGrockit Gets A $7 Million Venture Infusion And Launches Video Q&A Site Grockit AnswersGorgeous Photos, Tablet Browsing: 500px Debuts New iPad AppSamsung Galaxy Nexus, HTC Vigor To Launch November 10?Freelance.com: Facebook App, 3D, HTML5, And Cocoa Jobs On The RiseiPhone 4S First Weekend Sales Exceeds 4 Million, Doubles The Pace Of The iPhone 4Wahanda Secures $5.5 Million From Fidelity Growth Partners EuropeLook Out Uber: GroundLink Launches An Affordable, Mobile Private Car Service For New YorkersVideo Collaboration Software Maker ViVu Acquired By PolycomWith 400,000 Users Under Its Belt, SohoOS Plans Major Revamp5 Product Innovations From CEATEC 2011 In Japan (Video Gallery)Digital Media Companies Inuvo And Vertro To MergeRIM Apologizes With Free Apps & Technical Support For Three Days Of DowntimeCMU Researchers Turn Any Surface Into A TouchscreenPosted: 17 Oct 2011 09:14 AM PDT";

        JET_INSTANCE instance;
        JET_SESID sesid;
        JET_DBID dbid;
        JET_TABLEID tableid;

        JET_COLUMNDEF columndef = new JET_COLUMNDEF();

        // Initialize ESENT. Setting JET_param.CircularLog to 1 means ESENT will automatically
        // delete unneeded logfiles. JetInit will inspect the logfiles to see if the last
        // shutdown was clean. If it wasn't (e.g. the application crashed) recovery will be
        // run automatically bringing the database to a consistent state.
        Api.JetCreateInstance(out instance, "instance");
        Api.JetSetSystemParameter(instance, JET_SESID.Nil, JET_param.CircularLog, 1, null);
        Api.JetInit(ref instance);
        Api.JetBeginSession(instance, out sesid, null, null);

        // Create the database. To open an existing database use the JetAttachDatabase and 
        // JetOpenDatabase APIs.
        Api.JetCreateDatabase(sesid, "edbtest.db", null, out dbid, CreateDatabaseGrbit.OverwriteExisting);

        // Create the table. Meta-data operations are transacted and can be performed concurrently.
        // For example, one session can add a column to a table while another session is reading
        // or updating records in the same table.
        // This table has no indexes defined, so it will use the default sequential index. Indexes
        // can be defined with the JetCreateIndex API.
        Api.JetBeginTransaction(sesid);
        Api.JetCreateTable(sesid, dbid, "table", 0, 100, out tableid);

        JET_COLUMNID id;
        columndef.coltyp = JET_coltyp.Binary;
        columndef.cp = JET_CP.ASCII;
        Api.JetAddColumn(sesid, tableid, "id", columndef, null, 0, out id);

        JET_COLUMNID blob;
        columndef.coltyp = JET_coltyp.LongBinary;
        //columndef.cp = JET_CP.ASCII;
        Api.JetAddColumn(sesid, tableid, "blob", columndef, null, 0, out blob);

        string indexDef = "+id\0\0";
        Api.JetCreateIndex(sesid, tableid, "primary", CreateIndexGrbit.IndexPrimary, indexDef, indexDef.Length, 100);
        //Api.JetSetCurrentIndex(sesid, tableid, null);
        Api.JetCommitTransaction(sesid, CommitTransactionGrbit.LazyFlush);

        long Process_MemoryStart = 0;
        Process MyProcess = System.Diagnostics.Process.GetCurrentProcess();
        Process_MemoryStart = MyProcess.PrivateMemorySize64;
        Console.WriteLine("Before loop : " + Process_MemoryStart / 1024 + "KB");

        int i = 0;
        for (int t = 0; t < 20; t++)
        {
            Api.JetBeginTransaction(sesid);
            for (int j = 0; j < 500; j++)
            {
                i = t * 500 + j;
                string dataString = techcrunchString + i.ToString();

                byte[] data = Encoding.UTF8.GetBytes(dataString);
                string keyString = i.ToString();
                byte[] key = Encoding.UTF8.GetBytes(keyString);

                //store
                Api.MakeKey(sesid, tableid, key, MakeKeyGrbit.NewKey);
                bool exists = Api.TrySeek(sesid, tableid, SeekGrbit.SeekEQ);

                if (exists)
                {
                    Api.JetPrepareUpdate(sesid, tableid, JET_prep.ReplaceNoLock);
                    //Console.WriteLine("store: " + "update");
                }
                else
                {
                    Api.JetPrepareUpdate(sesid, tableid, JET_prep.Insert);
                    Api.SetColumn(sesid, tableid, id, key);
                    //Console.WriteLine("store: " + "insert");
                }
                Api.SetColumn(sesid, tableid, blob, data);
                Api.JetUpdate(sesid, tableid);

                if (i % 500 == 0)
                {
                    long Process_MemoryStart1 = 0;
                    Process MyProcess1 = System.Diagnostics.Process.GetCurrentProcess();
                    Process_MemoryStart1 = MyProcess1.PrivateMemorySize64;
                    Console.WriteLine("Finished " + i.ToString() + " : " + Process_MemoryStart1 / 1024 + "KB");

                }

            }
            Api.JetCommitTransaction(sesid, CommitTransactionGrbit.None);


        }
        Process_MemoryStart = 0;
        MyProcess = System.Diagnostics.Process.GetCurrentProcess();
        Process_MemoryStart = MyProcess.PrivateMemorySize64;
        Console.WriteLine("Loop finished: " + Process_MemoryStart / 1024 + "KB");

        // Terminate ESENT. This performs a clean shutdown.
        Api.JetCloseTable(sesid, tableid);
        Process_MemoryStart = 0;
        MyProcess = System.Diagnostics.Process.GetCurrentProcess();
        Process_MemoryStart = MyProcess.PrivateMemorySize64;
        Console.WriteLine("After close table: " + Process_MemoryStart / 1024 + "KB");

        Api.JetEndSession(sesid, EndSessionGrbit.None);
        Process_MemoryStart = 0;
        MyProcess = System.Diagnostics.Process.GetCurrentProcess();
        Process_MemoryStart = MyProcess.PrivateMemorySize64;
        Console.WriteLine("After end session: " + Process_MemoryStart / 1024 + "KB");

        Api.JetTerm(instance);

        Process_MemoryStart = 0;
        MyProcess = System.Diagnostics.Process.GetCurrentProcess();
        Process_MemoryStart = MyProcess.PrivateMemorySize64;
        Console.WriteLine("After term instance: " + Process_MemoryStart / 1024 + "KB");
    }

In above code, It goes up to about 100 MB. Only if I do Api.JetTerm(instance), the memory is freed.

In my real problem, I have to constantly insert rows of large amount of data for many many times, so it won't work for me in this way as the memory will be eventually eaten up.

Could anyone please help me with this?

**Why esent holds the memory even if I commited the transaction?

I am suspecting it is the undo things inside esent which holds the memory, and if it is, how to turn it off? I don't need undo thing.**

Thanks

P.S: I have tried this test() method in both 32bit & 64bit Windows, both has the same memory problem.

Answer 1

Would this be of any help: http://www.nikosbaxevanis.com/bonus-bits/2010/10/adventures-using-rhino-servicebus.html ?

Microsoft.Isam.Esent.Interop.JET_param, CacheSizeMax This parameter configures the maximum size of the database page cache. The size is in database pages. If this parameter is left to its default value, then the maximum size of the cache will be set to the size of physical memory when JetInit is called.

Setting the Microsoft.Isam.Esent.Interop.SystemParameters.CacheSizeMax to 1024 or 512 seems to solve the problem with the increasing memory usage.

Answer 2

Sorry I'm so late to the party...

I'd note that doing this is not fixing a memory leak, it's reducing performance of the engine. The cache size is caching database pages in memory such that you hit the disk less...just like a good database should. ;) If you reduce this you're hurting perf and requiring more disk IO.

The database engine is wise enough to back off when it sees other people using memory, so in theory you shouldn't have the size the cache like this, it will release memory if it sees others wanting it.

So, net, unless you really know you need to do this, I wouldn't recommend it.