Serialization error during parfor

Question

I am working on a project that involves 2D projections of a specimen and i am require to transform it into cross-sectional image and then stack each cross sections to get a 3D image. I using Matlab for my simulations but as it involves huge data sets the processing time is very huge. I want that reduced significantly. Here is my code-

 projection_length = 1600;


 images = cell(1,500);


 for i = 1:500
    fname= sprintf('pre%03d.tif', i);
    images{i} = imread(fname);
 end

 parfor q = 1:projection_length
   tmpData = zeros(1600, 500); 
   for i = 1:500
     fname= sprintf('pre%03d.tif', i);
     tmpData(:, i) = images{i}(1 : 1600, q, :);

     disp(['Analyzing projection ' num2str(q)  ' of ' num2str(projection_length) ', Angle ' num2str(i) '...']);
   end
   idata=255-tmpData;   
   H = iradon(idata, 0.72, 'Hann', 0.8, 1600 ); 
   postfname= sprintf('post%06d.tif', q);
   imwrite(H, postfname);

 end

I am loading all images into a cell in the beginning because I found that imread used most of the processing time when I read specific regions again and again in the loop, but since the images are large and the images variable is also too large in this case, i run out of memory. For projection length of around 1600 I undergo a serialization error that says-

Warning: Error caught during construction of remote parfor code. The parfor construct will now be run locally rather than on the remote matlabpool. The most likely cause of this is an inability to send over input arguments because of a serialization error. The error report from the caught error is: Error using distcompserialize Out of Memory during serialization

Error in distcomp.remoteparfor (line 36) obj.SerializedInitData = distcompMakeByteBufferHandle(distcompserialize(varargin));

Error in parallel_function (line 437) P = distcomp.remoteparfor(W, @make_channel, parfor_C);

Error in Microfab_bead_reconstruction (line 15) parfor q = 1:projection_length

In parallel_function at 450

Is there a way to fix this, is it due to the huge memory of the variable 'images' which results in communication overhead when used parallely? Are there any other alternatives? I tried using the distributed class of variables but then I won't be able to use the parfor.

Answer 1

The problem of your code is, you are using a broadcast variable image which means the variable is copied to each worker. Instead of creating a cell array, create a big 4-D matrx containing all images in a stack. If you now use it using tmp=images(:,:,q,:) inside your parfor loop, which returns all the data one iteration of the parfor loop needs, only that really used part of images is copied to each worker. It is now a properly sliced variable.