SYSTEM AND METHOD FOR TEXT NORMALIZATION IN NOISY CHANNELS

• US 20160335244A1
• Filed: 12/28/2015
• Published: 11/17/2016
• Est. Priority Date: 05/14/2015
• Status: Active Grant

• Current Assignee: Nice Ltd
• Original Assignee: Nice Systems Limited (Nice Ltd)
• Inventors: IZSAK, Peter, ACHLOW, Inna, SHAFRAN, Victor, WEISMAN, Hila
• Granted Patent: US 10,803,241 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 40/157 using dictionaries or tables
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/232 Orthographic correction, e.g. spell checking or vowelisation
• CPC Class Codes: G06F 40/242 Dictionaries
• CPC Class Codes: G06F 40/253 Grammatical analysis; Style critique
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/30 Semantic analysis

Abstract

Systems and methods for text normalization in a plurality of noisy channels receive a text entry and channel origin data of the text entry; determine whether the text entry matches an in-vocabulary (IV) entry or whether the text entry is an out-of-vocabulary (OOV) entry; if the text entry is determined to have a matching IV entry, output the matching IV entry, and if the text entry is determined to be an OOV entry, implement a channel-specific error-type adapter framework based on the channel origin data, wherein the channel-specific error-type adapter framework is optimized for a specific channel from which the text entry originated; normalize the text entry using the channel-specific error-type adapter framework; and output one or more candidate normalized forms of the text entry.

Claims

1. A method for text normalization in a plurality of noisy channels, performed on a computing device having a processor, memory, and one or more code sets stored in the memory and executing in the processor, the method comprising:

receiving, by the processor, a text entry and channel origin data of the text entry;

determining, by the processor, whether the text entry matches an in-vocabulary (IV) entry or whether the text entry is an out-of-vocabulary (OOV) entry;

if the text entry is determined to have a matching IV entry;

outputting, by the processor, the matching IV entry; and

if the text entry is determined to be an OOV entry;

implementing, by the processor, a channel-specific error-type adapter framework based on the channel origin data;

wherein the channel-specific error-type adapter framework is optimized for a specific channel from which the text entry originated;

normalizing, by the processor, the text entry using the channel-specific error-type adapter framework; and

outputting one or more candidate normalized forms of the text entry.

2. The method of claim 1:

wherein the channel-specific error-type adapter framework comprises one or more error-type adapters; and

wherein each error-type adaptor is configured to model a different type of error to be normalized.

3. The method of claim 1, comprising an initial step of building an interpolated language model and one or more static lexicons to be implemented when normalizing the text entry using the channel-specific error-type adapter framework.

4. The method of claim 3, wherein the interpolated language model comprises a merged combination of a basic language model and one or more channel-specific language models.

5. The method of claim 3, wherein the one or more static lexicons comprise at least one of a proper name lexicon, an abbreviation lexicon, and an acronym lexicon.

6. The method of claim 2, wherein the one or more error-type adapters comprise at least one of a spelling error-type adapter, an abbreviation error-type adapter, an acronym error-type adapter, a phonetic shorthand error-type adapter, and a word concatenation error-type adapter.

7. The method of claim 6, wherein the spelling error-type adapter is generated using one or more matrices, each matrix computing a probability of a specific edit operation being performed to identify a spelling error correction.

8. The method of claim 2, wherein the channel-specific error-type adapter framework further comprises a probabilistic model of one or more error-type priors.

9. The method of claim 2, further comprising applying linguistic heuristics to expand the channel-specific error-type adapter framework.

10. A system for text normalization in a plurality of noisy channels, comprising:

a processor;

a memory; and

one or more code sets stored in the memory and executing in the processor, which, when executed, configure the processor to;

receive a text entry and channel origin data of the text entry;

determine whether the text entry matches an in-vocabulary (IV) entry or whether the text entry is an out-of-vocabulary (OOV) entry;

if the text entry is determined to have a matching IV entry;

output the matching IV entry; and

if the text entry is determined to be an OOV entry;

implement a channel-specific error-type adapter framework based on the channel origin data;

wherein the channel-specific error-type adapter framework is optimized for a specific channel from which the text entry originated;

normalize the text entry using the channel-specific error-type adapter framework; and

output one or more candidate normalized forms of the text entry.

11. The system of claim 10:

wherein the channel-specific error-type adapter framework comprises one or more error-type adapters; and

wherein each error-type adaptor is configured to model a different type of error to be normalized.

12. The system of claim 10, wherein the one or more code sets further configure the processor to build an interpolated language model and one or more static lexicons to be implemented when normalizing the text entry using the channel-specific error-type adapter framework.

13. The system of claim 12, wherein the interpolated language model comprises a merged combination of a basic language model and one or more channel-specific language models.

14. The system of claim 12, wherein the one or more static lexicons comprise at least one of a proper name lexicon, an abbreviation lexicon, and an acronym lexicon.

15. The system of claim 11, wherein the one or more error-type adapters comprise at least one of a spelling error-type adapter, an abbreviation error-type adapter, an acronym error-type adapter, a phonetic shorthand error-type adapter, and a word concatenation error-type adapter.

16. The system of claim 15, wherein the one or more code sets configure the processor to generate the spelling error-type adapter using one or more matrices, each matrix computing a probability of a specific edit operation being performed to identify a spelling error correction.

17. The system of claim 11, wherein the channel-specific error-type adapter framework further comprises a probabilistic model of one or more error-type priors.

18. The system of claim 11, further comprising applying linguistic heuristics to expand the channel-specific error-type adapter framework.

19. A method for text normalization in a plurality of noisy channels, performed on a computing device having a processor, memory, and one or more code sets stored in the memory and executing in the processor, the method comprising:

receiving, by the processor, a text entry and channel origin data of the text entry;

determining, by the processor, whether the text entry matches an in-vocabulary (IV) entry or whether the text entry is an out-of-vocabulary (OOV) entry;

wherein the matching IV entry is outputted when the text entry is determined to have a matching IV entry,

wherein a channel-specific error-type adapter framework is implemented based on the channel origin data when the text entry is determined to be an OOV entry, and

wherein the channel-specific error-type adapter framework is optimized for a specific channel from which the text entry originated;

normalizing, by the processor, the text entry using the channel-specific error-type adapter framework; and

outputting one or more candidate normalized forms of the text entry.

20. The method of claim 19:

wherein the channel-specific error-type adapter framework comprises one or more error-type adapters; and

wherein each error-type adaptor is configured to model a different type of error to be normalized.