RWWTokenizer(3C++)

Name
     RWWTokenizer - Rogue Wave library class

Synopsis
              #include <rw/wtoken.h>



              RWWString str("a string of tokens", RWWString::ascii);
          RWWTokenizer(str);  // Lex the above string




Description
     Class RWWTokenizer is designed to break a string up into separate tokens,
     delimited by arbitrary "white space."  It can be thought of as an
     iterator for strings and as an alternative to the C library function
     wstok() which has the unfortunate side effect of changing the string
     being tokenized.

Persistence
     None

Example
              #include <rw/wtoken.h>
          #include <rw/rstream.h>
          main(){
            RWWString a(L"Something is rotten in the state of Denmark");
            RWWTokenizer next(a);   // Tokenize the string a
            RWWString token;        // Will receive each token
            // Advance until the null string is returned:
            while (!(token=next()).isNull())
              cout << token << "0;
          }


     Program output:

                   Something
               is
               rotten
               in
               the
               state
               of
               Denmark

Public Constructor
              RWWTokenizer(const RWWString& s);
     Construct a tokenizer to lex the string s.

Public Member Function
              RWWSubString
          operator();


     Advance to the next token and return it as a substring.  The tokens are
     delimited by any of the four wide characters in L" 0. (space, tab,
     newline and null).

              RWWSubString
          operator()(const wchar_t* s);


     Advance to the next token and return it as a widesubstring.  The tokens
     are  delimited by any wide character in s, or any embedded wide null.

              RWWSubString
          operator()(const wchar_t* s,size_t num);


     Advance to the next token and return it as a substring.  The tokens are
     delimited by any of the first num wide characters in s.  Buffer s may
     contain embedded nulls, and must contain at least num wide characters.
     Tokens will not be delimited by nulls unless s contains nulls.