python pandas multi-index assignment gives key error

993 views Asked by At

in a loop, I fill a multi-indexed dataframe with elements using the loc[(x,y)] for indexing. While this works well for some thousand elements, I suddenly get a key-error.

# setupt of empty dataframe with multi-index
df = pd.DataFrame()
df.index = pd.MultiIndex.from_tuples((), names=['cycle','itemNr']) 

# adding data elements using loc in a loop like this
df.loc[ (currentCycle, currentItem) , 'a' ] = 99

After adding 10000 rows, each with 15 columns, I get a key error.

Am I using the wrong index method?

Do I need to ensure sorted multi-index?

Is there a size limit?

df.shape
OUT: (10000, 15)

df.loc[(282,1), 'columnA'] = 99 
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-40-0a92a62250dd> in <module>()
----> 1 df.loc[(282,1), 'columnA'] = 99

C:\Anaconda3\lib\site-packages\pandas\core\indexing.py in __setitem__(self, key, value)
    192             key = com._apply_if_callable(key, self.obj)
    193         indexer = self._get_setitem_indexer(key)
--> 194         self._setitem_with_indexer(indexer, value)
    195 
    196     def _has_valid_type(self, k, axis):

C:\Anaconda3\lib\site-packages\pandas\core\indexing.py in _setitem_with_indexer(self, indexer, value)
    373                     self.obj.is_copy = None
    374 
--> 375                     nindexer.append(labels.get_loc(key))
    376 
    377                 else:

C:\Anaconda3\lib\site-packages\pandas\core\indexes\multi.py in get_loc(self, key, method)
   2089             key = _values_from_object(key)
   2090             key = tuple(map(_maybe_str_to_time_stamp, key, self.levels))
-> 2091             return self._engine.get_loc(key)
   2092 
   2093         # -- partial selection or non-unique index

pandas\_libs\index.pyx in pandas._libs.index.MultiIndexHashEngine.get_loc()

pandas\_libs\index.pyx in pandas._libs.index.MultiIndexHashEngine.get_loc()

pandas\_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.MultiIndexHashTable.get_item()

pandas\_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.MultiIndexHashTable.get_item()

KeyError: (282, 1)

How to Reproduce:

CVS file reproduce_loc_assignment_error.csv http://s000.tinyupload.com/index.php?file_id=61488607596539177759

The file has 10000 lines and just 3 columns (2 index columns and one data column a)

df = pd.read_csv('reproduce_loc_assign_error.csv', index_col=[0,1])
df.loc[(282,2),'a'] = 100 #( gives key error)
0

There are 0 answers