METHOD AND SYSTEM FOR GENERATING NAMED ENTITIES

• US 20180336192A1
• Filed: 06/30/2017
• Published: 11/22/2018
• Est. Priority Date: 05/18/2017
• Status: Active Grant

• Current Assignee: Wipro Limited
• Original Assignee: Wipro Limited
• Inventors: JAGAN, Balaji, KUMAR NANJAPPA, Naveen
• Granted Patent: US 10,467,346 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06F 40/53 Processing of non-Latin text kana-to-kanji conversion G06F40/129; vowelisation G06F40/232

Abstract

This disclosure relates generally to natural language processing, and more particularly to system and method for generating named entities. In one embodiment, a method is provided for generating named entities. The method includes extracting a plurality of named entities in a primary language from a plurality of digital content in the primary language, transliterating each of the plurality of named entities in the primary language to a set of possible named entities in a secondary language, determining a correct named entity in the secondary language from among the set of possible named entities in the secondary language, and generating a named entity in a subsequent secondary language corresponding to the correct named entity in the secondary language. It should be noted that the plurality of named entities in the primary language are named entities in the subsequent secondary language, and the subsequent secondary language is related to the secondary language.

Claims

1. A method of generating named entities, the method comprising:

extracting, by a named entity generation engine, a plurality of named entities in a primary language from a plurality of digital content in the primary language;

transliterating, by the named entity generation engine, each of the plurality of named entities in the primary language to a set of possible named entities in a secondary language;

determining, by the named entity generation engine, a correct named entity in the secondary language from among the set of possible named entities in the secondary language; and

generating, by the named entity generation engine, a named entity in a subsequent secondary language corresponding to the correct named entity in the secondary language, wherein the plurality of named entities in the primary language are named entities in the subsequent secondary language, and wherein the subsequent secondary language is related to the secondary language.

2. The method of claim 1, wherein the plurality of named entities in the primary language is extracted using a named entity extraction model.

3. The method of claim 2, wherein the named entity extraction model is trained by manually tagging an initial plurality of named entities in the primary language in an initial plurality of digital content in the primary language.

4. The method of claim 3, wherein the initial plurality of digital content in the primary language are curated from across a plurality of genres, and are relevant to a population versant with the subsequent secondary language.

5. The method of claim 1, wherein each of the plurality of named entities comprise at least one of a person, a place, and an Organization.

6. The method of claim 1, wherein each of the plurality of named entities in the primary language is transliterated to the set of possible named entities in the secondary language using a plurality of predefined transliteration frameworks, and wherein each of the plurality of predefined transliteration frameworks comprise a plurality of pre-defined mapping tables in Unicode values.

7. The method of claim 6, wherein each of the plurality of predefined transliteration frameworks comprises at least one of a Harvard-Kyoto transliteration framework, an ISO 15919 transliteration framework, and a simplified customized standard transliteration framework, and wherein each of the plurality of pre-defined mapping tables comprises at least one of a vowels mapping table, a constants mapping table, and a matras mapping table.

8. The method of claim 1, wherein transliteration further comprises:

retrieving a sequence of symbols for the primary language for each of the plurality of named entities in the primary language;

retrieving a sequence of symbols for the secondary language corresponding to the sequence of symbols for the primary language; and

combining the sequence of symbols in the secondary language to generate the set of possible named entities in the secondary language.

9. The method of claim 1, wherein the correct named entity in the secondary language is determined from among the set of possible named entities in the secondary language using a long short term memory (LSTM) model based on a confidence score.

10. The method of claim 9, wherein the LSTM model is trained with a large dataset comprising of a plurality of named entities of the secondary language and a plurality of corresponding transliterated named entities in the primary language.

11. The method of claim 1, wherein the named entity in the subsequent secondary language corresponding to the correct named entity in the secondary language is generated using a multi-lingual character level tree model.

12. The method of claim 1, wherein the primary language is English, wherein the secondary language is Hindi, and the subsequent secondary language is one of a plurality of Indian languages.

13. A system for generating named entities, the system comprising:

at least one processor; and

a computer-readable medium storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising;

extracting a plurality of named entities in a primary language from a plurality of digital content in the primary language;

transliterating each of the plurality of named entities in the primary language to a set of possible named entities in a secondary language;

determining a correct named entity in the secondary language from among the set of possible named entities in the secondary language; and

generating a named entity in a subsequent secondary language corresponding to the correct named entity in the secondary language, wherein the plurality of named entities in the primary language are named entities in the subsequent secondary language, and wherein the subsequent secondary language is related to the secondary language.

14. The system of claim 13, wherein the plurality of named entities in the primary language is extracted using a named entity extraction model.

15. The system of claim 14, wherein the named entity extraction model is trained by manually tagging an initial plurality of named entities in the primary language in an initial plurality of digital content in the primary language, and wherein the initial plurality of digital content in the primary language are curated from across a plurality of genres and are relevant to a population versant with the subsequent secondary language.

16. The system of claim 13, wherein each of the plurality of named entities in the primary language is transliterated to the set of possible named entities in the secondary language using a plurality of predefined transliteration frameworks, and wherein each of the plurality of predefined transliteration frameworks comprise a plurality of pre-defined mapping tables in Unicode values.

17. The system of claim 16, wherein each of the plurality of predefined transliteration frameworks comprises at least one of a Harvard-Kyoto transliteration framework, an ISO 15919 transliteration framework, and a simplified customized standard transliteration framework, and wherein each of the plurality of pre-defined mapping tables comprises at least one of a vowels mapping table, a constants mapping table, and a matras mapping table.

18. The system of claim 13, wherein transliteration further comprises:

retrieving a sequence of symbols for the primary language for each of the plurality of named entities in the primary language;

retrieving a sequence of symbols for the secondary language corresponding to the sequence of symbols for the primary language; and

combining the sequence of symbols in the secondary language to generate the set of possible named entities in the secondary language.

19. The system of claim 13, wherein the correct named entity in the secondary language is determined from among the set of possible named entities in the secondary language using a long short term memory (LSTM) model based on a confidence score, and wherein the LSTM model is trained with a large dataset comprising of a plurality of named entities of the secondary language and a plurality of corresponding transliterated named entities in the primary language.

20. The system of claim 13, wherein the named entity in the subsequent secondary language corresponding to the correct named entity in the secondary language is generated using a multi-lingual character level tree model.

21. A non-transitory computer-readable medium storing computer-executable instructions for:

extracting a plurality of named entities in a primary language from a plurality of digital content in the primary language;

transliterating each of the plurality of named entities in the primary language to a set of possible named entities in a secondary language;

determining a correct named entity in the secondary language from among the set of possible named entities in the secondary language; and

generating a named entity in a subsequent secondary language corresponding to the correct named entity in the secondary language, wherein the plurality of named entities in the primary language are named entities in the subsequent secondary language, and wherein the subsequent secondary language is related to the secondary language.