How to find large objects in ZODB

Question

I'm trying to analyze my ZODB because it grew really large (it's also large after packing).

The package zodbbrowser has a feature that displays the amount of bytes of an object. It does so by getting the length of the pickled state (name of the variable), but it also does a bit of magic which I don't fully understand.

How would I go to find the largest objects in my ZODB?

Answer 1

I've written a method which should do exactly this. Feel free to use it, but be aware that this is very memory consuming. The package zodbbrowser must be installed.

def zodb_objects_by_size(self):
    """
        Recurse over the ZODB tree starting from self.aq_parent. For
        each object, use zodbbrowser's implementation to get the raw
        object state. Put each length into a Counter object and
        return a list of the biggest objects, specified by path and
        size.
    """
    from zodbbrowser.history import ZodbObjectHistory
    from collections import Counter

    def recurse(obj, results):
        # Retrieve state pickle from ZODB, get length
        history = ZodbObjectHistory(obj)
        pstate = history.loadStatePickle()
        length = len(pstate)

        # Add length to Counter instance
        path = '/'.join(obj.getPhysicalPath())
        results[path] = length

        # Recursion
        for child in obj.contentValues():

            # Play around portal tools and other weird objects which
            # seem to contain themselves
            if child.contentValues() == obj.contentValues():
                continue

            # Rolling in the deep
            try:
                recurse(child, results)
            except (RuntimeError, AttributeError), err:
                import pdb; pdb.set_trace()  ## go debug

    results = Counter()
    recurse(self.aq_parent, results)
    return results.most_common()