How to use boost::tokenizer as a C++ class member?

Question

I am replacing RWCTokenizer with boost::tokenizer and I can't get it to compile as the code is currently designed. This is what its basically doing right now:

parse.h
    RWCTokenizer*           myTokenizer;

parse.C
function parse::A
    myTokenizer = new RWCTokenizer(myTokenLine);

function parse::B
    token =  ((*myTokenizer)());

function parse::C
    if(myTokenizer == EcDNull)
    {
        return ""; // end of tokens
    }

So basically they are setting up the parse in A, get tokens in B, and checking for the end in C.

Therefore I need to have iterator and tok variables declared in the include so they can be accessed by A, B, and C. But when I compile, the myTok says it needs 2 arguments so I don't know how to save that information. This is currently what I tried to do:

parse.h
    boost::tokenizer< boost::char_separator<char> > myTok;
    boost::tokenizer< boost::char_separator<char> >::iterator myTokenizer2;

parse.C
    function parse::A
    boost::char_separator<char> sep(" ");   // break on white space
    std::string myTokenLine_TMP = myTokenLine.data();
    boost::tokenizer< boost::char_separator<char> > tok(myTokenLine_TMP, sep);
    myTokenizer = tok.begin();
    myTok = tok;

    function parse::B
    RWCString token = *myTokenizer;
    myTokenizer++;

    function parse::C
    if (myTokenizer == myTok.end())
    {
        return "";   // end of tokens
    }

The only the examples I see online has the boost code in a c main function where its all together vs. a C++ class example. The only other thing right now I can think of is to declare the boost stuff a static at the top of the class which is something I really don't want to do.

If I can some how save the myTok.end() part I should be able to compile. But I think I need the hold structure because the line I am parse was passed into tok(myTokenLine_TMP, sep) and it should go out of scope when I leave the function so saving the end() part would be useless.

So how do you do it in a C++ class?

Answer 1

To do it in a C++ class, you just need to make the tokenizer a member of your class, like

class MyClass
{
private:
     boost::tokenizer<boost::char_seperator<char>> m_tokenizer;
};

Answer 2

I have decided to coded a helper class that is using strtok vs. boost tokenizer that will allow me to duplicate the Roguewave tokenizer class.