Open topic with navigation

# Prepare Text

It is important to ensure that the text used for custom language model building is cleaned up to a reasonable level.

- Remove any material that does not occur in standard written English, such as HTML tags, and anything that would not usually occur in spoken language, such as tables.
- Ensure that sentence breaks (periods) are present in the text.
- Ensure that there are no duplicated sections in the text.
- Ensure that the text encoding is UTF-8.

Note: Japanese, Korean, Mandarin, and Taiwanese Mandarin languages require text segmentation before HPE IDOL Speech Server can process them. Text segmentation inserts whitespace between words. The `LanguageModelBuild` task segments text if you set the `DoSegment` parameter to `True` (see Build the Language Model).

## Normalize Text

You must normalize the text used for language model building before processing so that word representations are standardized. For example, ‘1’ and ‘one’ are treated as two different representations of the same digit. For more information about the normalization scheme used in HPE IDOL Speech Server, see Audio Transcript Requirements.

Many of the HPE IDOL Speech Server text operations require you to normalize input text as an initial step. For details of the normalization procedure, see Run Text Normalization.

© 2013-2016 Hewlett Packard Enterprise Development LP | Send documentation feedback to HPE Close Send Documentation Feedback To open the configured email client on this computer, open an email window. Otherwise, copy the information below to a web mail client, and send this email to AutonomyTPFeedback@hpe.com. We appreciate your feedback! Product: Topic Title: Help Topic: Feedback:

_HP_HTML5_bannerTitle.htm