Boost C++ Libraries of the most highly regarded and expertly designed C++ library projects in the world. Herb Sutter and Andrei Alexandrescu, C++ Coding Standards

This is the documentation for an old version of boost. Click here for the latest Boost documentation.

C++ Boost

Tokenizer Class

  template <
        class TokenizerFunc = char_delimiters_separator<char>, 
        class Iterator = std::string::const_iterator,
        class Type = std::string
  class tokenizer

The tokenizer class provides a container view of a series of tokens contained in a sequence. You set the sequence to parse and the TokenizerFunction to use to parse the sequence either upon construction or using the assign member function. Note: No parsing is actually done upon construction. Parsing is done on demand as the tokens are accessed via the iterator provided by begin.


// simple_example_1.cpp

int main(){
   using namespace std;
   using namespace boost;
   string s = "This is,  a test";
   tokenizer<> tok(s);
   for(tokenizer<>::iterator beg=tok.begin(); beg!=tok.end();++beg){
       cout << *beg << "\n";

The output from simple_example_1 is:


Template Parameters

Parameter Description
TokenizerFunc The TokenizerFunction used to parse the sequence.
Iterator The type of the iterator the specifies the sequence.
Type The type of the token, typically string.


Related Types



iterator The type returned by begin and end. Note: the category of iterator will be at most ForwardIterator. It will be InputIterator if the Iterator template parameter is an InputIterator. For any other category, it will be ForwardIterator.
const_iterator Same type as iterator.
value_type Same type as the template parameter Type
reference Same type as value_type&
const_reference Same type as const reference
pointer Same type as value_type*
const_pointer Same type as const pointer
size_type void
difference_type void


Construction and Member Functions

tokenizer(Iterator first, Iterator last,const TokenizerFunc& f = TokenizerFunc()) 

template<class Container>
tokenizer(const Container& c,const TokenizerFunc& f = TokenizerFunc())

void assign(Iterator first, Iterator last)

void assign(Iterator first, Iterator last, const TokenizerFunc& f)

template<class Container>
void assign(const Container& c)

template<class Container>
void assign(const Container& c, const TokenizerFunc& f)

iterator begin() const 

iterator end() const



c A container that contains the sequence to parse. Note: c.begin() and c.end() must be convertible to the template parameter Iterator.
f A functor that is a model of TokenizerFunction that will be used to parse the sequence.
first The iterator that represents the beginning position in the sequence to be parsed.
last The iterator that represents the past the end position in the sequence to be parsed.


Valid HTML 4.01 Transitional

Revised 16 February, 2008

Copyright © 2001 John R. Bandela

Distributed under the Boost Software License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at