Detokenizer User's Guide

User's Guide for the Detokenizers.

Detokenizer User's Guide

The StrFile created by the Tokenizer contains the tokenized file and all of the extracted strings. Plus, by the time it gets to the Detokenizer, it contains the translation of the extracted string. The Detokenizer replaces each token in the tokenized file with the translation string thus completing the translation process for that file.

Like the Tokenizers, the Detokenizers are written for specific file types and styles. If a Tokenizer extension is needed then an extended Detokenizer is probably needed also. Detokenizers are provided for the following types of files:

File Type Detokenizer Class
Java Properties com.g11ntoolkit.detokenizer.DetokPropertyResourceBundle
Java List Resource Bundle com.g11ntoolkit.detokenizer.DetokResourceBundle
Java Server Page com.g11ntoolkit.detokenizer.DetokJSPFile
HTML com.g11ntoolkit.detokenizer.DetokHTMLFile
Cascading Style Sheet com.g11ntoolkit.detokenizer.DetokCSSFile
JavaScript com.g11ntoolkit.detokenizer.DetokJSFile
Resource Catalog com.g11ntoolkit.detokenizer.DetokRC
Structured Query Language (SQL) Data Definition Language (DDL) or Data Manipulation Language (DML) com.g11ntoolkit.detokenizer.DetokSQLFile
Message Catalog com.g11ntoolkit.detokenizer.DetokMCFile
XML com.g11ntoolkit.detokenizer.DetokXMLFile
XSL com.g11ntoolkit.detokenizer.DetokXSLFile

If a file is not one of these then chances are that there is no Detokenizer for it. If a file type that is not in the list is required, a Detokenizer extension can be implemented to handle the file correctly.

Each detokenizer is written as an independent tool that can be executed from the command line. To make things easier the process control (L10NProcess...) Ant files are available and contain Ant targets to determine when to use each of the detokenizers based on the contents of the Project Contol File. All detokenizers are controlled by targets in the L10NProcess-detoktasks.xml file.

Return to:   Top of page

by Bill Rich