org.mpxj.common

Class Tokenizer

java.lang.Object

org.mpxj.common.Tokenizer

Direct Known Subclasses:

InputStreamTokenizer, ReaderTokenizer

public abstract class Tokenizer
extends Object

This class implements a tokenizer based loosely on java.io.StreamTokenizer. This tokenizer is designed to parse records from an MPX file correctly. In particular, it will handle empty fields, represented by adjacent field delimiters.

Field Summary

Fields

Modifier and Type	Field and Description
static int	TT_EOF
static int	TT_EOL
static int	TT_WORD

Constructor Summary

Constructors

Constructor and Description
Tokenizer()

Method Summary

Modifier and Type	Method and Description
String	getToken() This method retrieves the text of the last token found.
int	getType() This method retrieves the type of the last token found.
int	nextToken() This method retrieves the next token and returns a constant representing the type of token found.
protected abstract int	read() This method must be implemented to read the next character from the data source.
void	setDelimiter(char delimiter) This method is used to set the delimiter character recognised by the tokenizer.
protected boolean	startQuotedIsValid(StringBuilder buffer) This method allows us to control the behaviour of the tokenizer for quoted text.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TT_EOL

public static final int TT_EOL

See Also:

Constant Field Values

TT_EOF

public static final int TT_EOF

See Also:

Constant Field Values

TT_WORD

public static final int TT_WORD

See Also:

Constant Field Values

Constructor Detail

Tokenizer

public Tokenizer()

Method Detail

read

protected abstract int read()
                     throws IOException

This method must be implemented to read the next character from the data source.

Returns:

next character

Throws:

IOException

nextToken

public int nextToken()
              throws IOException

This method retrieves the next token and returns a constant representing the type of token found.

Returns:

token type value

Throws:

IOException

startQuotedIsValid

protected boolean startQuotedIsValid(StringBuilder buffer)

This method allows us to control the behaviour of the tokenizer for quoted text. Normally quoted text begins with a quote character at the first position within a field. As this method is protected, subclasses can alter this behaviour if required.

Parameters:

buffer - the field contents read so far

Returns:

true if it is valid to treat the subsequent text as quoted

getToken

public String getToken()

This method retrieves the text of the last token found.

Returns:

last token text

getType

public int getType()

This method retrieves the type of the last token found.

Returns:

last token type

setDelimiter

public void setDelimiter(char delimiter)

This method is used to set the delimiter character recognised by the tokenizer.

Parameters:

delimiter - delimiter character