Transferring Large Data from Web Worker to Main Thread in Vue.js

Question

I am facing challenges in efficiently transferring a large dataset (potentially more than a million items) generated in a Web Worker to the main thread in my Vue.js application. Currently, when I try to pass the generated grid from the worker to the main thread, it leads to errors such as "maximum call stack size exceeded" or even crashes the page.

Here's a simplified version of my current Web Worker code:

// gridWorker.js
self.onmessage = function (e) {
  const { isUpdateColor, row, col, cellSize } = e.data;

  if (isUpdateColor) {
    const grid = Array.from({ length: row }, (_, i) => {
      return Array.from({ length: col }, (_, j) => {
        return {
          i,
          j,
          code: `spot_${i}_${j}`,
          // ... other properties
        };
      });
    });

    self.postMessage({ grid });
  }
};

//main thread
<v-stage>
    <v-layer ref="pathFinderLayer">
        <div v-if="showTravelDistancePage">
          <template v-for="rows in grid">
            <v-rect v-for="(item, index) in rows" :key="item.code + '-' + index" :config="item"
              @click="gridClick(item)" />
          </template>

        </div>
      </v-layer>
</v-stage>


methods:
prepareGrid(isUpdateColor,cleanPath=true) {
    this.gridWorker = new GridWorker();
      this.gridWorker.onmessage = (event) => {
        const { grid } = event.data;
          this.grid = grid;
        }

      };
}

I suspect the issue arises from trying to pass such a large dataset to the main thread. Is there a more efficient way to handle this situation? Even when I pass partial data chunk by chunk, its still crash my page if the grid object is too many.

Approach 1: Generating on Main Thread

Initially, I attempted to generate the large grid directly on the main thread. However, this approach led to page crashes due to the sheer volume of objects.

Approach 2: Generating in Web Worker and Passing Entire Grid

To address the performance issues, I moved the grid generation process to a Web Worker. While this successfully offloaded the generation to a separate thread, passing the entire generated grid back to the main thread still resulted in page crashes.

Approach 3: Generating in Web Worker Chunk by Chunk

In the latest attempt, I opted to generate the grid in chunks within the Web Worker. I then passed each chunk individually to the main thread, storing the chunks inside the main grid. Unfortunately, this strategy also led to crashes at a certain point.

Answer 1

Seems XY problem. A short answer - you shouldn't do such things as generating a million items UI data. You should only generate data visible on the screen, that's called called a virtual scrolling. So you generate you data on the fly when needed.

If you still need to generate 1 million items ahead, do that in a worker and transfer only visible data on scrolling.

An example with a simple virtual list (it would even scroll much better). As you see I generate HTML on the fly (a more advanced scrolling would offset the top item imitating scrolling within an item too):

const arr = Array.from({length:1000000}, () => Math.random());

unorderedList.style.height = (arr.length * 32) + 'px';

drawItems();

wrapper.addEventListener('scroll', drawItems);

function drawItems(){
  const pos = wrapper.scrollTop;
  const from = pos / 32 | 0;
  const to = from + wrapper.offsetHeight / 32 | 0 + 1;
  unorderedList.innerHTML = arr.slice(from, to).map((item, idx) => `<li>${from + idx + 1}: ${item}</li>`).join('');
  let top = -32;
  [...unorderedList.children].forEach((li, idx) => li.style.top = `${top += 32}px`);
}

#wrapper{
  max-height:100vh;
  overflow:auto;
}
body{
margin:0;
padding:0;
overflow:hidden;
}
li{
 position:absolute;
 height:32px;
 line-hieght:32px;
}

<div id="wrapper">
<ul id="unorderedList"></ul>
</div>