HYBRID ADAPTATION OF NAMED ENTITY RECOGNITION

• US 20140163951A1
• Filed: 12/07/2012
• Published: 06/12/2014
• Est. Priority Date: 12/07/2012
• Status: Abandoned Application

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Nikoulina, Vassilina, Sandor, Agnes
• Application Number:
  US13/707,745
• Publication Number:
  US 20140163951A1
• Time in Patent Office: Days
• US Class Current: 704/4
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06F 40/42 Data-driven translation

Abstract

A machine translation method includes receiving a source text string and identifying any named entities. The identified named entities may be processed to exclude common nouns and function words. Features are extracted from the source text string relating to the identified named entities. Based on the extracted features, a protocol is selected for translating the source text string. A first translation protocol includes forming a reduced source string from the source text string in which the named entity is replaced by a placeholder, translating the reduced source string by machine translation to generate a translated reduced target string, while processing the named entity separately to be incorporated into the translated reduced target string. A second translation protocol includes translating the source text string by machine translation, without replacing the named entity with the placeholder. The target text string produced by the selected protocol is output.

Claims

1. A machine translation method comprising:

receiving a source text string in a source language;

identifying named entities in the source text string;

optionally, processing the identified named entities to exclude at least one of common nouns and function words from the named entities;

extracting features from the optionally processed source text string relating to the identified named entities;

with a processor, for at least one of the named entities, based on the extracted features, selecting a protocol for translating the source text string, the protocol being selected from a plurality of translation protocols,

a first of the translation protocols including;

forming a reduced source string from the source text string in which the named entity is replaced by a placeholder;

translating the reduced source string by machine translation to generated a translated reduced target string,

processing the named entity separately, and

incorporating the processed named entity into the translated reduced target string to produce a target text string in the target language;

a second of the translation protocols including;

translating the source text string by machine translation, without replacing the named entity with the placeholder, to produce a target text string in the target language; and

outputting the target text string produced by the selected protocol.

2. The method of claim 1, wherein the features include features selected from the group consisting of:

a frequency of the named entity in training data;

a confidence in the translation of a named entity dictionary used in the separate processing of the named entity;

a feature defined by a context of the named entity in the source string;

a probability of the named entity in the context, estimated on a source corpus;

a probability of the placeholder replacing the named entity in the context, estimated on a source corpus;

and combinations thereof.

3. The method of claim 1, wherein selecting of the translation protocol comprises, with a prediction model, predicting which of the translation protocols would yield a better translation.

4. The method of claim 3, further comprising training the predictive model on features extracted for named entities of a set of training source strings, and a comparison of a translation score for each source string in which the named entity is replaced by a placeholder and translated separately and a translation score for the respective source string without the replacement.

5. The method of claim 1, wherein the method includes the processing of the identified named entities to exclude at least one of common nouns and function words from the named entities.

6. The method of claim 5, wherein the processing of the identified named entities to exclude at least one of common nouns and function words from the named entities is performed with a set of predefined rules.

7. The method of claim 1, wherein the identifying of named entities in the source string comprises identifying a type of the named entity, the type of named entity being selected from a predefined set of named entity types and wherein the placeholder comprises the identified type of named entity.

8. The methods of claim 7, wherein the set of named entity types includes named entity types selected from the group consisting of PERSON, ORGANIZATION, LOCATION, and DATE.

9. The method of claim 7, wherein the translating of the reduced source string by machine translation to generated the translated reduced target string comprises translating the reduced source string with a machine translation model which has been trained on a parallel training corpus of source and target text strings in which at least some of the named entities are replaced by placeholders.

10. The method of claim 9, wherein some of the named entities in the parallel training corpus are not replaced by placeholders to produce a hybrid translation model which is adapted for translation of the source string in both protocols.

11. The method of claim 1, wherein the machine translation comprises phrase-based statistical machine translation.

12. A computer program product comprising a non-transitory medium storing instructions, which when executed by a computer, perform the method of claim 1.

13. A system comprising memory storing instructions for performing the method of claim 1 and a processor, in communication with the memory, for executing the instructions.

14. A machine translation system comprising:

a named entity recognition component for identifying named entities in an input source text string in a source language;

optionally, a rule applying component which applies rules for processing the identified named entities to exclude at least one of common nouns and function words from the named entities;

a feature extraction component for extracting features from the optionally processed source text string relating to the identified named entities;

a prediction component for selecting a translation protocol for translating the source string based on the extracted features, the translation protocol being selected from a set of translation protocols including a first translation protocol in which the named entity is replaced by a placeholder to form a reduced source string, the reduced source string being translated separately from the named entity, and a second translation protocol in which the source text string is translated without replacing the named entity with the placeholder, to produce a target text string in the target language; and

a machine translation component for performing the selected translation protocol; and

a processor for implementing at least one of the components.

15. The system of claim 14, wherein the prediction component inputs the features to a prediction model for predicting whether the translation would be better if the named entity were to be replaced by a placeholder and translated separately.

16. The system of claim 14, comprising the rule applying component.

17. A method for forming a machine translation system comprising:

optionally, providing rules for processing named entities identified in a source text string to exclude at least one of common nouns and function words from the named entities;

with a processor, learning a prediction model for predicting a suitable translation protocol from a set of translation protocols for translating the optionally processed source text string, the learning comprising;

for each of a training set of optionally processed source text strings;

extracting features from the optionally processed source text strings relating to the identified named entities, and

for each of the translation protocols, computing a translation score for a target text string generated by the translation protocol; and

learning the prediction model based on the extracted features and translation scores;

providing a prediction component which applies the model to features extracted from the optionally processed source text string to select one of the translation protocols.

18. The method of claim 17, wherein a first of the translation protocols includes:

forming a reduced source string from the source text string in which the named entity is replaced by a placeholder;

translating the reduced source string by machine translation to generated a translated reduced target string,

processing the named entity separately, and

incorporating the processed named entity into the translated reduced target string to produce a target text string in the target language; and

a second of the translation protocols includes;

translating the source text string by machine translation, without replacing the named entity with the placeholder, to produce a target text string in the target language.

19. The method of claim 17, further generating the rules from a training set of source sentences in which named entities have been recognized.

20. The method of claim 17, further comprising generating a machine translation system for the translating of the reduced source string, the generating comprising:

providing a parallel training corpus of pairs of source and target text strings, at least some of the pair in the parallel corpus including a reduced source string in which at least one named entity is replaced with a placeholder and a reduced target string in which a corresponding named entity is replaced with a placeholder; and

learning the machine translation system using the parallel training corpus.

21. The method of claim 20, wherein for at least a fraction of the named entities there is no placeholder replacement to provide a hybrid machine translation system for translating a source string which contains both a placeholder and an original named entity.

22. A computer program product comprising a non-transitory medium storing instructions, which when executed by a computer, perform the method of claim 17.

23. A system comprising memory storing instructions for performing the method of claim 17 and a processor, in communication with the memory, for executing the instructions.