Handling extremely large and sparse matrices in RcppArmadillo

Question

I'm trying to pass a diagonal matrix W to an Rcpp function. The problem is that W has size 1,000,000 by 1,000,000, which is (I think) well outside the limits Armadillo allows (even when using a C++11 compiler with ARMA_64BIT_WORD enabled).

Since W is a diagonal matrix, it is extremely sparse. For that reason, I first generated a dense representation of W (using the Matrix package function Diagonal). I then passed these compressed representation of W to my function. I thought this would solve any memory problems. Here's a small example:

C++ Code:

#define ARMA_64BIT_WORD 1
#include <RcppArmadillo.h>
// [[Rcpp::depends(RcppArmadillo)]]
// [[Rcpp::plugins(cpp11)]]

using namespace Rcpp;
using namespace arma;


// [[Rcpp::export]]
int test(sp_mat W){
  
  return 0;
  
}

R Code:

# define the diagonal matrix
nrows <- 1e6
W <- Matrix::Diagonal(nrows)

# call Rcpp function
test(W)

However, despite using a compressed representation, I still get this error.

error: SpMat::init(): requested size is too large; suggest to compile in C++11 mode and/or enable ARMA_64BIT_WORD

Is there any way to handle W so that I can pass it into test and perform matrix operations?

A similar question was asked here. However, I think the solution provided by @Dirk worked in that case since the input matrix was still small compared to my W.

Answer 1

The originally posted code was actually correct, but somehow the machine it was running on was not---see the discussion above for details.

As an add-on, here is a slightly edited version of the code without global namespace, actual access to the matrix (passing was enough, this is simply more explicit) and returning void. We also added the usual 'call it from R for me' trick one can do with Rcpp Attributes.

Code

// lightly edited version of question, working fine for me
#define ARMA_64BIT_WORD 1
#include <RcppArmadillo.h>
// [[Rcpp::depends(RcppArmadillo)]]

// [[Rcpp::export]]
void spmatTest(arma::sp_mat M) {
  // show a simple access of matrix,
  // here we just sum elements on diagonal
  Rcpp::Rcout << "Sum of diag is "
              << arma::as_scalar(arma::sum(M.diag()))
              << std::endl;
}

/*** R
spmatTest( Matrix::Diagonal( 1e6 ))
spmatTest( Matrix::Diagonal( 1e7 ))
spmatTest( Matrix::Diagonal( 1e8 ))
*/

Output

R> Rcpp::sourceCpp("~/git/stackoverflow/64428776/answer.cpp")

R> spmatTest( Matrix::Diagonal( 1e6 ))
Sum of diag is 1e+06

R> spmatTest( Matrix::Diagonal( 1e7 ))
Sum of diag is 1e+07

R> spmatTest( Matrix::Diagonal( 1e8 ))
Sum of diag is 1e+08
R>