A utility class for splitting strings into tokens based on delimiters. More...

#include <easy3d/util/tokenizer.h>

Public Member Functions
	Tokenizer (const char src, const char delims=nullptr)
	Constructor that initializes the tokenizer with a source string and optional delimiters.

void	setConsumeWS (bool ws)
	Sets whether to consume whitespace characters.

bool	isWhitespace (const char c)
	Checks if the given character is a whitespace character.

bool	isQuote (const char c)
	Checks if the given character is a quote character.

bool	isEOL (const char c)
	Checks if the given character is an end-of-line character.

bool	isTerm (const char c)
	Checks if the given character is a termination character.

bool	isDelim (const char c)
	Checks if the given character is a delimiter.

bool	atEOF ()
	Checks if the tokenizer has reached the end of the source string.

char	consumeWhitespace ()
	Consumes whitespace characters from the source string.

char	consumeOneDelim ()
	Consumes one delimiter character from the source string.

void	consumeToEOL ()
	Consumes characters up to the end of the line.

bool	readToken ()
	Reads the next token from the source string.

bool	requireToken (const char *find)
	Checks if the next token matches the given string.

bool	requireTokenDelim (const char *find)
	Checks if the next token matches the given string and consumes a delimiter.

char	getTermChar () const
	Gets the character that caused the stop of the last token read.

bool	getLastToken (std::string &returnTok)
	Gets the last read token as a std::string.

const char *	getLastTokenPtr ()
	Gets the last read token as a const char*.

uint32_t	getLastTokenLen () const
	Gets the length of the last read token.

bool	getTokenString (std::string &returnTok)
	Gets the next token as a std::string.

bool	getTokenString (char out[], const uint32_t outmax)
	Gets the next token as a char array with a maximum size.

bool	getTokenFloat (float &out)
	Gets the next token as a floating-point number.

uint32_t	getTokenFloatArray (float out[], uint32_t size)
	Gets the next tokens as an array of floating-point numbers.

uint32_t	getTokenIntArray (int32_t out[], uint32_t size)
	Gets the next tokens as an array of integer numbers.

bool	getTokenInt (int32_t &out)
	Gets the next token as an integer.

bool	getTokenUint (uint32_t &out)
	Gets the next token as an unsigned integer.

bool	getTokenBool (bool &out)
	Gets the next token as a boolean value.

Detailed Description

A utility class for splitting strings into tokens based on delimiters.

This class provides methods to tokenize a given string based on a set of delimiters. It can handle empty tokens and allows for retrieving individual tokens by index.

Constructor & Destructor Documentation

◆ Tokenizer()

Tokenizer	(	const char *	src,
		const char *	delims = nullptr )

inlineexplicit

Constructor that initializes the tokenizer with a source string and optional delimiters.

Parameters

src	The source string to tokenize.
delims	The delimiters to use for tokenizing. If nullptr, default delimiters are used.

Member Function Documentation

◆ atEOF()

bool atEOF ( )

inline

Checks if the tokenizer has reached the end of the source string.

Returns: True if the tokenizer has reached the end of the source string, false otherwise.

◆ consumeOneDelim()

char consumeOneDelim ( )

inline

Consumes one delimiter character from the source string.

Returns: The delimiter character consumed.

◆ consumeWhitespace()

char consumeWhitespace ( )

inline

Consumes whitespace characters from the source string.

Returns: The last whitespace character consumed.

◆ getLastToken()

bool getLastToken ( std::string & returnTok )

inline

Gets the last read token as a std::string.

Parameters

returnTok The string to store the last read token.

Returns: True if the token was successfully retrieved, false otherwise.

◆ getLastTokenLen()

uint32_t getLastTokenLen ( ) const

inline

Gets the length of the last read token.

Returns: The length of the last read token.

◆ getLastTokenPtr()

const char * getLastTokenPtr ( )

inline

Gets the last read token as a const char*.

Returns: The last read token.

◆ getTermChar()

char getTermChar ( ) const

inline

Gets the character that caused the stop of the last token read.

Returns: The termination character.

◆ getTokenBool()

bool getTokenBool ( bool & out )

inline

Gets the next token as a boolean value.

Parameters

out	The boolean value to store the next token.

Returns: True if the token was successfully retrieved, false otherwise.

◆ getTokenFloat()

bool getTokenFloat ( float & out )

inline

Gets the next token as a floating-point number.

Parameters

out	The floating-point number to store the next token.

Returns: True if the token was successfully retrieved, false otherwise.

◆ getTokenFloatArray()

uint32_t getTokenFloatArray	(	float	out[],
		uint32_t	size )

inline

Gets the next tokens as an array of floating-point numbers.

Parameters

out	The array to store the floating-point numbers.
size	The size of the array.

Returns: The number of elements read.

◆ getTokenInt()

bool getTokenInt ( int32_t & out )

inline

Gets the next token as an integer.

Parameters

out	The integer to store the next token.

Returns: True if the token was successfully retrieved, false otherwise.

◆ getTokenIntArray()

uint32_t getTokenIntArray	(	int32_t	out[],
		uint32_t	size )

inline

Gets the next tokens as an array of integer numbers.

Parameters

out	The array to store the integer numbers.
size	The size of the array.

Returns: The number of elements read.

◆ getTokenString() [1/2]

bool getTokenString	(	char	out[],
		const uint32_t	outmax )

inline

Gets the next token as a char array with a maximum size.

Parameters

out	The char array to store the next token.
outmax	The maximum size of the char array.

Returns: True if the token was successfully retrieved, false otherwise.

◆ getTokenString() [2/2]

bool getTokenString ( std::string & returnTok )

inline

Gets the next token as a std::string.

Parameters

returnTok The string to store the next token.

Returns: True if the token was successfully retrieved, false otherwise.

◆ getTokenUint()

bool getTokenUint ( uint32_t & out )

inline

Gets the next token as an unsigned integer.

Parameters

out	The unsigned integer to store the next token.

Returns: True if the token was successfully retrieved, false otherwise.

◆ isDelim()

bool isDelim ( const char c )

inline

Checks if the given character is a delimiter.

Parameters

c	The character to check.

Returns: True if the character is a delimiter, false otherwise.

◆ isEOL()

bool isEOL ( const char c )

inline

Checks if the given character is an end-of-line character.

Parameters

c	The character to check.

Returns: True if the character is an end-of-line character, false otherwise.

◆ isQuote()

bool isQuote ( const char c )

inline

Checks if the given character is a quote character.

Parameters

c	The character to check.

Returns: True if the character is a quote character, false otherwise.

◆ isTerm()

bool isTerm ( const char c )

inline

Checks if the given character is a termination character.

Parameters

c	The character to check.

Returns: True if the character is a termination character, false otherwise.

◆ isWhitespace()

bool isWhitespace ( const char c )

inline

Checks if the given character is a whitespace character.

Parameters

c	The character to check.

Returns: True if the character is a whitespace character, false otherwise.

◆ readToken()

bool readToken ( )

inline

Reads the next token from the source string.

Returns: True if a token was successfully read, false otherwise.

◆ requireToken()

bool requireToken ( const char * find )

inline

Checks if the next token matches the given string.

Parameters

find	The string to match.

Returns: True if the next token matches the given string, false otherwise.

◆ requireTokenDelim()

bool requireTokenDelim ( const char * find )

inline

Checks if the next token matches the given string and consumes a delimiter.

Parameters

find	The string to match.

Returns: True if the next token matches the given string and a delimiter was consumed, false otherwise.

◆ setConsumeWS()

void setConsumeWS ( bool ws )

inline

Sets whether to consume whitespace characters.

Parameters

ws	True to consume whitespace, false otherwise.

The documentation for this class was generated from the following file:

G:/3_code/Easy3D/easy3d/util/tokenizer.h

Public Member Functions

Detailed Description

Constructor & Destructor Documentation

◆ Tokenizer()

Member Function Documentation

◆ atEOF()

◆ consumeOneDelim()

◆ consumeWhitespace()

◆ getLastToken()

◆ getLastTokenLen()

◆ getLastTokenPtr()

◆ getTermChar()

◆ getTokenBool()

◆ getTokenFloat()

◆ getTokenFloatArray()

◆ getTokenInt()

◆ getTokenIntArray()

◆ getTokenString() [1/2]

◆ getTokenString() [2/2]

◆ getTokenUint()

◆ isDelim()

◆ isEOL()

◆ isQuote()

◆ isTerm()

◆ isWhitespace()

◆ readToken()

◆ requireToken()

◆ requireTokenDelim()

◆ setConsumeWS()