I keep getting "ValueError: Length of values (1) does not match length of index (11)" even though I am following an example problem

Question

I am attempting to couple a pymoo optimization algorithm with a regression model. I found this fantastic example on github, and am attempting to follow the steps starting from In[16]. The example problem is probably using an outdated method of loading the pickle file (it wasn't working for me), so I'm using Model= load_model('RegModel') for my first model output and similarlyModel2=load_model('Reg2Model') for my second output. I was able to follow the rest of the example up to In[69]. I'm also using two different pkl files to predict two different outcomes of my data, but when I use

result=pd.DataFrame(list(res.X))
result['Output1']=res.F
result['Output2']=Model2.predict(result)

I get the error in the question title, namely:ValueError: Length of values (1) does not match length of index (11) (note that my length is 11 instead of the example's 5 because I'm using a different dataset. The methodology however is the same).

I tried not using the pd.DataFrame object and pass it as a numpy list but that also didn't work. At this point I'm stumped. If I don't define Output 2 and simply define Output 1 alone Output=res.F my code compiles, but it's obviously nonsensical.

Any help is appreciated, including the intuition behind In[69]. Appended is the full error message:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[14], line 4
      1 result= pd.DataFrame(list(res.X))
      2 result
----> 4 result['Output1']= res.F
      5 result['Output2']=Reg2Model.predict(result)

File ~\AppData\Local\Programs\Python\Python311\Lib\site-packages\pandas\core\frame.py:3980, in DataFrame.__setitem__(self, key, value)
   3977     self._setitem_array([key], value)
   3978 else:
   3979     # set column
-> 3980     self._set_item(key, value)

File ~\AppData\Local\Programs\Python\Python311\Lib\site-packages\pandas\core\frame.py:4174, in DataFrame._set_item(self, key, value)
   4164 def _set_item(self, key, value) -> None:
   4165     """
   4166     Add series to DataFrame in specified column.
   4167 
   (...)
   4172     ensure homogeneity.
   4173     """
-> 4174     value = self._sanitize_column(value)
   4176     if (
   4177         key in self.columns
   4178         and value.ndim == 1
   4179         and not is_extension_array_dtype(value)
   4180     ):
   4181         # broadcast across multiple columns if necessary
   4182         if not self.columns.is_unique or isinstance(self.columns, MultiIndex):

File ~\AppData\Local\Programs\Python\Python311\Lib\site-packages\pandas\core\frame.py:4915, in DataFrame._sanitize_column(self, value)
   4912     return _reindex_for_setitem(Series(value), self.index)
   4914 if is_list_like(value):
-> 4915     com.require_length_match(value, self.index)
   4916 return sanitize_array(value, self.index, copy=True, allow_2d=True)

File ~\AppData\Local\Programs\Python\Python311\Lib\site-packages\pandas\core\common.py:571, in require_length_match(data, index)
    567 """
    568 Check the length of data matches the length of the index.
    569 """
    570 if len(data) != len(index):
--> 571     raise ValueError(
    572         "Length of values "
    573         f"({len(data)}) "
    574         "does not match length of index "
    575         f"({len(index)})"
    576     )

ValueError: Length of values (1) does not match length of index (11)

Answer 1

You're error is due to trying to add an array of length 1 to a dataframe which has a size of 11 (length of index is 11).

You should check how res.X and res.F look like. Are they consistent? And how does the dataframe look like? Is there something else in the dataframe?

Usually the amount of solutions in the solution space res.F should equal the amount of sets in the design space res.X.

Answer 2

UPDATE: The problem was that the genetic algorithm number of generations was too low, which somehow meant that the algorithm terminated before it could run to its entirety. Upon increasing it to 1000 instead of 100, it ran and this error disappeared. I have another error now, but this question has been solved.