libs/spirit/doc/lex/tokenizing.qbk
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:lexer_tokenizing Tokenizing Input Data]
[heading The tokenize function]
The `tokenize()` function is a helper function simplifying the usage of a lexer
in a stand alone fashion. For instance, you may have a stand alone lexer where all
that functional requirements are implemented inside lexer semantic actions.
A good example for this is the [@../../example/lex/word_count_lexer.cpp word_count_lexer]
described in more detail in the section __sec_lex_quickstart_2__.
[wcl_token_definition]
The construct used to tokenize the given input, while discarding all generated
tokens is a common application of the lexer. For this reason __lex__ exposes an
API function `tokenize()` minimizing the code required:
// Read input from the given file
std::string str (read_from_file(1 == argc ? "word_count.input" : argv[1]));
word_count_tokens<lexer_type> word_count_lexer;
std::string::iterator first = str.begin();
// Tokenize all the input, while discarding all generated tokens
bool r = tokenize(first, str.end(), word_count_lexer);
This code is completely equivalent to the more verbose version as shown in the
section __sec_lex_quickstart_2__. The function `tokenize()` will return either
if the end of the input has been reached (in this case the return value will
be `true`), or if the lexer couldn't match any of the token definitions in the
input (in this case the return value will be `false` and the iterator `first`
will point to the first not matched character in the input sequence).
The prototype of this function is:
template <typename Iterator, typename Lexer>
bool tokenize(Iterator& first, Iterator last, Lexer const& lex
, typename Lexer::char_type const* initial_state = 0);
[variablelist where:
[[Iterator& first] [The beginning of the input sequence to tokenize. The
value of this iterator will be updated by the
lexer, pointing to the first not matched
character of the input after the function
returns.]]
[[Iterator last] [The end of the input sequence to tokenize.]]
[[Lexer const& lex] [The lexer instance to use for tokenization.]]
[[Lexer::char_type const* initial_state]
[This optional parameter can be used to specify
the initial lexer state for tokenization.]]
]
A second overload of the `tokenize()` function allows specifying of any arbitrary
function or function object to be called for each of the generated tokens. For
some applications this is very useful, as it might avoid having lexer semantic
actions. For an example of how to use this function, please have a look at
[@../../example/lex/word_count_lexer.cpp word_count_functor.cpp]:
[wcf_main]
Here is the prototype of this `tokenize()` function overload:
template <typename Iterator, typename Lexer, typename F>
bool tokenize(Iterator& first, Iterator last, Lexer const& lex, F f
, typename Lexer::char_type const* initial_state = 0);
[variablelist where:
[[Iterator& first] [The beginning of the input sequence to tokenize. The
value of this iterator will be updated by the
lexer, pointing to the first not matched
character of the input after the function
returns.]]
[[Iterator last] [The end of the input sequence to tokenize.]]
[[Lexer const& lex] [The lexer instance to use for tokenization.]]
[[F f] [A function or function object to be called for
each matched token. This function is expected to
have the prototype: `bool f(Lexer::token_type);`.
The `tokenize()` function will return immediately if
`F` returns `false.]]
[[Lexer::char_type const* initial_state]
[This optional parameter can be used to specify
the initial lexer state for tokenization.]]
]
[/heading The generate_static_dfa function]
[endsect]
