com.g11ntoolkit.tokenizer
Class FileTokenizer

java.lang.Object
  |
  +--com.g11ntoolkit.tokenizer.FileTokenizer

Direct Known Subclasses:

TokCSSFile, TokJSFile, TokListResourceBundle, TokMC, TokPropertyResourceBundle, TokRC, TokSQLFile, TokXMLFile

public class FileTokenizer

extends java.lang.Object

General class for a file tokenizer.

This is intended to be a class whose sub-classes take care of all the work. This class is here to provide a first level of interpretation as to with which kind of a file we are dealing.

Version:

2005/06/27

Author:

Bill Rich, Wilandra Consulting LLC
Copyright © 2001-2005, Wilandra Consulting LLC. All rights reserved.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

See License Agreement.

Field Summary

protected static java.util.Hashtable	context The context hashtable.
protected static java.lang.String	encoding The encoding specifies which encoding is used for the input file.
protected static java.lang.StringBuffer	inFileBuffer The string buffer which will contain the input file data for the subclasses to use.
protected static java.util.ResourceBundle	jspVars JSP variables Resource Bundle.
private static java.util.logging.Logger	log The log used for all messages from this class.
protected static java.util.ResourceBundle	mrb Message Resource Bundle.
protected static java.lang.String	product The name of the Product that contains the input file.
protected static java.lang.String	productVersion The version for the product that contains the input file.
protected static java.util.Hashtable	revContext The Tokens associated with their contexts.
protected java.util.Locale	targetLocale The target locale specifies which locale is used for the resource bundle.
protected static TokContext	tokContext The context information.
protected static Token	token The token is used to get the unique token numbers.
protected static java.util.ResourceBundle	vrb Variables Resource Bundle.

Constructor Summary

	FileTokenizer() Establishes an instance of the class for public use in order to determine which subclass to use.
protected	FileTokenizer(java.lang.String anEncoding) Used by the subclasses to establish an instance of the class and save the encoding information for use in processing the file.
protected	FileTokenizer(java.lang.String anEncoding, java.util.Locale aLocale, Token aToken) Used by the subclasses to establish an instance of the class and save all the input information for use in processing the file.

Method Summary

protected static java.lang.String	cleanConcats(java.lang.String val) Removes any string concats from the specified string.
static java.lang.String	getEncoding() Returns the encoding name for the input file.
java.lang.StringBuffer	getFileBuffer() Returns the file buffer that contains the input file contents.
TokContext	getKey(Token token) Returns the key for a specified token in the tokenized file.
static java.lang.String	getProductName() Returns the product name.
static java.lang.String	getProductVersion() Returns the product version.
Token	getToken(TokContext key) Returns the token for a specified key in the tokenized file.
java.util.Enumeration	keys() Returns a list of keys in the tokenized file.
static void	readFile(java.lang.String inFileName) Reads in the specified file and stores it in the string buffer.
static void	readFilePlain(java.lang.String inFileName, java.lang.String encoding) Reads in the specified file and stores it in the string buffer.
void	setContext(TokContext ctxt, Token tok) Saves the association between a context in the tokenized file and its token.
static void	setEncoding(java.lang.String e) Sets the encoding name for the input file.
static void	setProductName(java.lang.String p) Sets the product name.
static void	setProductVersion(java.lang.String v) Sets the product version.
protected void	setupTok(java.lang.String anEncoding, java.util.Locale aLocale) Used by the subclasses to set up the common part of the checking of any DSP file.
protected static void	setupTok(java.lang.String inFileName, java.lang.String contextFileName) Used by the subclasses to set up the common part of the tokenizing of any file and reads the file into the string buffer.
java.util.Enumeration	tokens() Returns a list of tokens in the tokenized file.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

log

private static java.util.logging.Logger log

The log used for all messages from this class.

mrb

protected static java.util.ResourceBundle mrb

Message Resource Bundle.

vrb

protected static java.util.ResourceBundle vrb

Variables Resource Bundle.

jspVars

protected static java.util.ResourceBundle jspVars

JSP variables Resource Bundle.

token

protected static Token token

The token is used to get the unique token numbers.

encoding

protected static java.lang.String encoding

The encoding specifies which encoding is used for the input file.

The encoding will also be used on the output files.

targetLocale

protected java.util.Locale targetLocale

The target locale specifies which locale is used for the resource bundle.

inFileBuffer

protected static java.lang.StringBuffer inFileBuffer

The string buffer which will contain the input file data for the subclasses to use.

tokContext

protected static TokContext tokContext

The context information.

context

protected static java.util.Hashtable context

The context hashtable.

This is a table of the contexts in the file associated with their tokens.

revContext

protected static java.util.Hashtable revContext

The Tokens associated with their contexts.

This is a table of the tokens in the file associated with their contexts (i.e., a reverse context).

product

protected static java.lang.String product

The name of the Product that contains the input file.

productVersion

protected static java.lang.String productVersion

The version for the product that contains the input file.

Constructor Detail

FileTokenizer

public FileTokenizer()

Establishes an instance of the class for public use in order to determine which subclass to use.

FileTokenizer

protected FileTokenizer(java.lang.String anEncoding,
                        java.util.Locale aLocale,
                        Token aToken)
                 throws FileTokenizerError

Used by the subclasses to establish an instance of the class and save all the input information for use in processing the file.

Parameters:

anEncoding - a String specifying the encoding to use on the input file

aLocale - a Locale object

aToken - a Token object

Throws:

FileTokenizerError - when anything goes wrong

FileTokenizer

protected FileTokenizer(java.lang.String anEncoding)
                 throws FileTokenizerError

Used by the subclasses to establish an instance of the class and save the encoding information for use in processing the file.

Parameters:

anEncoding - a String specifying the encoding to use on the input file

Throws:

FileTokenizerError - when anything goes wrong

Method Detail

setupTok

protected void setupTok(java.lang.String anEncoding,
                        java.util.Locale aLocale)
                 throws FileTokenizerError

Used by the subclasses to set up the common part of the checking of any DSP file.

Parameters:

anEncoding - a String specifying the encoding to use on the input file

aLocale - a Locale object

Throws:

FileTokenizerError - when anything goes wrong

setupTok

protected static void setupTok(java.lang.String inFileName,
                               java.lang.String contextFileName)
                        throws FileTokenizerError

Used by the subclasses to set up the common part of the tokenizing of any file and reads the file into the string buffer.

Parameters:

inFileName - a String specifying the input file name

contextFileName - a String specifying the file name to use for the token id and context

Throws:

FileTokenizerError - when anything goes wrong

readFile

public static void readFile(java.lang.String inFileName)
                     throws FileTokenizerError

Reads in the specified file and stores it in the string buffer.

Coverts any non-ascii characters to their unicode value.

Parameters:

inFileName - a String specifying the name of the file to read

Throws:

FileTokenizerError - for any errors

readFilePlain

public static void readFilePlain(java.lang.String inFileName,
                                 java.lang.String encoding)
                          throws FileTokenizerError

Reads in the specified file and stores it in the string buffer.

Parameters:

inFileName - a String specifying the name of the file to read

encoding - a String specifying the encoding for the input file

Throws:

FileTokenizerError - for any errors

cleanConcats

protected static java.lang.String cleanConcats(java.lang.String val)
                                        throws FileTokenizerError

Removes any string concats from the specified string.

This replaces each string concat with a single blank and returns the resulting string. The length of the specified string must be longer than 3 characters before this method will do anything.

A string concat is assumed to be a double quote, followed by some amount of white space, followed by a plus sign (+), followed by some amount of white space, followed by a double quote. The amount of white space can be none in both places. This is why the minimum length of the string is checked before trying to remove the concats.

Parameters:

val - a String specifying the string to clean

Returns:

a String representing the cleaned string

Throws:

FileTokenizerError - when anything goes wrong

keys

public java.util.Enumeration keys()

Returns a list of keys in the tokenized file.

Returns:

an Enumeration object containing the context objects in this object or null if there are no context objects in this object

tokens

public java.util.Enumeration tokens()

Returns a list of tokens in the tokenized file.

Returns:

an Enumeration object containing the tokens in this object or null if there are no tokens in the object

getToken

public Token getToken(TokContext key)

Returns the token for a specified key in the tokenized file.

Parameters:

key - a TokContext object specifying the context

Returns:

a Token object representing the token for the specified context

getKey

public TokContext getKey(Token token)

Returns the key for a specified token in the tokenized file.

Parameters:

token - a Token object specifying the token

Returns:

a TokContext object representing the context for the specified token

setContext

public void setContext(TokContext ctxt,
                       Token tok)

Saves the association between a context in the tokenized file and its token.

Parameters:

ctxt - a TokContext object specifying the context for the specified token

tok - a Token object specifying the token for the specified context

getFileBuffer

public java.lang.StringBuffer getFileBuffer()

Returns the file buffer that contains the input file contents.

Returns:

a StringBuffer object representing the contents of the input file

setEncoding

public static void setEncoding(java.lang.String e)

Sets the encoding name for the input file.

Parameters:

e - a String specifying the encoding name

See Also:

getEncoding()

getEncoding

public static java.lang.String getEncoding()

Returns the encoding name for the input file.

Returns:

a String representing the encoding name

See Also:

setEncoding(java.lang.String)

setProductName

public static void setProductName(java.lang.String p)

Sets the product name.

Parameters:

p - a String specifying the name of the product

See Also:

getProductName()

getProductName

public static java.lang.String getProductName()

Returns the product name.

Returns:

a String representing the name of the product

See Also:

setProductName(java.lang.String)

setProductVersion

public static void setProductVersion(java.lang.String v)

Sets the product version.

Parameters:

v - a String specifying the version of the product

See Also:

getProductVersion()

getProductVersion

public static java.lang.String getProductVersion()

Returns the product version.

Returns:

a String representing the version of the product

See Also:

setProductVersion(java.lang.String)