I have a data.table with 57m records and 9 columns, one of which is causing a problem when I try to run some summary stats. The offending column is a factor with 3699 levels and I am receiveing an error from the following line of code:
> unique(da$UPC)
Error in unique.default(da$UPC): hash table is full
Now obviously I would just use: levels(da$UPC)
but I am trying to count the unique values which exist in each group as part of multiple j parameters/caluclations in a data.table group statement.
Interestingly unique(da$UPC[1:1000000])
works as expected however unique(da$UPC[1:10000000])
does not. Given that my table has 57m records this is an issue.
I tried converting the factor to a character and that works no problem as follows:
da$UPC = as.character(levels(da$UPC))[da$UPC]
unique(da$UPC)
Doing this does show me an additional "level" which is NA
. So because my data has some NAs in a factor column the unique function fails to work. I'm wondering if this is something which the developers are aware of an something which needs to be fixed? I found the following article on r-devel which might be relevant but I'm not sure and it does not mention data.table
.
Linked article: unique(1:3,nmax=1) freezes R!
sessionInfo:
R version 3.0.1 (2013-05-16)
Platform: x86_64-unknown-linux-gnu (64-bit)
locale:
[1] LC_CTYPE=C LC_NUMERIC=C
[3] LC_TIME=en_US.iso88591 LC_COLLATE=C
[5] LC_MONETARY=en_US.iso88591 LC_MESSAGES=en_US.iso88591
[7] LC_PAPER=C LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.iso88591 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] plyr_1.8 data.table_1.8.8
this snippet of code should place your missing observations into a regular level which will be more manageable to work with.
It sounds like you are ultimately trying to drop infrequent levels to assist in some sort of analysis. I wrote a function factorize() which I believe can help you. It buckets infrequent levels into an "Other" category.
Here's the link, please let me know if it helps.
[factorize()][1] https://github.com/greenpat/R-Convenience/blob/master/factorize.R
(reproduced below)