Handling Non-Serializable Object Attribute Checks in Parallel Processing with Python

25 views Asked by At

I'm facing a somewhat peculiar issue with Python and hoping for insights or solutions. Let's say I have a class containing a non-serializable object attribute (and changing this is not an option as the code is not mine), structured as follows:

class Class1:
    def __init__(self, non_serial_obj):
        self.non_serial_obj = non_serial_obj

class Class2:
    def __init__(self, obj_class1):
        self.obj_class1 = obj_class1

    def foo(self):
        return True

Additionally, Class2 includes a method named foo, which in this case simply returns True. So far, so good.

However, the problem arises when parallelizing the processing of multiple Class2 instances and checking for the existence of the foo attribute. An error occurs indicating that the object is not serializable due to the inability to communicate its code, although invoking the method itself does not cause any issues.

Is there a way to enable attribute checks to also work with this structure in a parallel processing context? To potentially aid in finding a solution, I'm using get_attr and env_method methods from the SubprocVecEnv class for invoking object methods in each process.

Any guidance or workaround would be greatly appreciated!

0

There are 0 answers