Package com.g11ntoolkit.tokenizer

The set of classes that make up the Tokenizer support for the G11NToolKit.

See:
Description

Class Summary

FileTokenizer	General class for a file tokenizer.
JSPTokenizer	Looks for the JSP language elements in a JSP or HTML file and protects them so that SDLX will process them correctly.
TokCSSFile	Tokenizes the specified buffer as a Cascading Style Sheet file.
TokJSFile	Tokenizes the specified buffer as a JavaScript file.
TokListResourceBundle	A subclass of FileTokenizer to tokenize the specified input file as though it were a ListResourceBundle file.
TokMC	Tokenize an MC file to extract the translatable strings.
TokPropertyResourceBundle	Tokenize the specified input file as though it were a PropertyResourceBundle file.
TokRC	RC file adapter for FileTokenizer.
TokSQLFile	Tokenizes the specified buffer as an SQL file.
TokXMLFile	Tokenize the specified buffer as an XML file.

Exception Summary

FileTokenizerError	Reports errors during tokenizing of a file.
FileTokRunError	Indicates there was some error during processing.
TokResourceBundleError	Reports errors during initialization of the tokenizer.

Package com.g11ntoolkit.tokenizer Description

The set of classes that make up the Tokenizer support for the G11NToolKit.

Package Specification

• The G11NToolKit Design Guide may contain some information on how the classes are intended to be used.

Related Documentation

• The Localization Process Guide contains information on the intended process. This will give some insight into the steps in the process and where the tokenization functions are used.