Systematic mass normalization of international titles

• US 10,678,827 B2
• Filed: 02/26/2016
• Issued: 06/09/2020
• Est. Priority Date: 02/26/2016
• Status: Active Grant

• Current Assignee: Workday Incorporated
• Original Assignee: Workday Incorporated
• Inventors: Giverts, Vladimir, Namjoshi, Parag Avinash, Boob, Pavan, Gateley, Kristy, Fan, Xiao, Au, Michael
• Primary Examiner(s): Ortiz Ditren, Belix M
• Assistant Examiner(s): Ohba, Mellissa M.
• Application Number:
  US15/055,287
• Publication Number:
  US 20170249383A1
• Time in Patent Office: 1,565 Days
• CPC Class Codes: G06F 16/214 Database migration support
• CPC Class Codes: G06F 16/3337 Translation of the query language, e.g. Chinese to English
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/289 Phrasal analysis, e.g. finite state techniques or chunking
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: G06Q 50/01 Social networking

Abstract

A system for generating a database of labeled foreign canonical titles includes an interface and a processor. The interface is to receive a title in a second language. The processor is to 1) store a set of n-grams in a first language in a first database; 2) sanitize the title into a sanitize title in the second language; 3) translate the sanitized title into a translated title in the first language; 4) break the translated title into n-grams; 5) determine labels for the n-grams using the first database; and 6) determine label to associate with the title.

Claims

1. A system for generating a database of labeled foreign titles comprising:

an interface to receive a title in a second language; and

a processor to;

store n-grams each with associated labels in a first language in a first database, wherein the first language and the second language are different;

sanitize the title in the second language into a sanitized title in the second language;

translate the sanitized title in the second language into a translated title in the first language;

break the translated title in the first language into intermediary n-grams in the first language;

determine parent n-grams of an intermediary n-gram of the intermediary n-grams, wherein the intermediary n-gram is a sub-string of the parent n-grams;

retrieve a set of labels associated with the parent n-grams using the first database, wherein the set of labels are in the first language; and

in response to determining that a matching threshold of a label of the set of labels is met, assign the label to the intermediary n-gram, wherein the matching threshold is a frequency the label occurs in the set of labels associated with the parent n-grams, wherein assigning the label comprises storing the label in the first language in a second database, and wherein the second database stores the title in the second language and the intermediary n-gram in the first language.

2. The system of claim 1, wherein the first database stores a set of n-grams associated with a first language title of a set of first language titles.

3. The system of claim 2, wherein the set of n-grams comprises a set of unigrams, a set of bi-grams, or a set of tri-grams.

4. The system of claim 3, wherein each unigram is one word in the first language title, each bi-gram is two contiguous words in the first language title;

and each tri-gram is three contiguous words in the first language title.

5. The system of claim 1, wherein the first database stores a set of attributes associated with an n-gram of the set of n-grams.

6. The system of claim 5, wherein the associated labels stored in the first database are assigned to an attribute of the set of attributes.

7. The system of claim 1, wherein the processor is further to generate a set of second language n-grams for the title in the second language.

8. The system of claim 7, wherein generating a set of second language n-grams for the title comprises:

generating a set of second language unigrams, wherein each second language unigram is one word in the title;

generating a set of second language bi-grams, wherein each second language bi-gram is two contiguous words in the title; and

generating a set of second language tri-grams, wherein each second language tri-gram is three contiguous words in the title.

9. The system of claim 1, wherein the second database stores a set of second language n-grams for the title in the second language.

10. The system of claim 9, wherein the second database stores a set of attributes associated with the title in the second language.

11. The system of claim 10, wherein assigned labels stored in the second database are associated with an attribute of the set of attributes.

12. The system of claim 9, wherein the second database stores a set of attributes associated with a second language n-gram of the set of second language n-grams.

13. The system of claim 12, wherein the assigned labels stored in the second database are associated with an attribute of the set of attributes.

14. The system of claim 1, wherein sanitizing the title comprises one or more of the following:

removing stop words, filtering characters, removing accents from characters, and tokenizing the title.

15. The system of claim 14, wherein tokenizing the title utilizes one or more of the following:

word boundaries, whitespace, and a dictionary.

16. The system of claim 1, wherein sanitizing the title comprises filtering tokens.

17. The system of claim 16, wherein filtering tokens comprises identifying articles or conjunctions.

18. The system of claim 1, wherein the second database stores the sanitized title in the second language.

19. A method for generating a database of labeled foreign titles comprising:

receiving a title in a second language;

storing n-grams each with associated labels in a first language in a first database, wherein the first language and the second language are different;

sanitizing, using a processor, the title in the second language into a sanitized title in the second language;

translating the sanitized title in the second language into a translated title in the first language;

breaking the translated title in the first language into intermediary n-grams in the first language;

determining parent n-grams of an intermediary n-gram of the intermediary n-grams, wherein the intermediary n-gram is a sub-string of the parent n-grams;

retrieving a set of labels associated with the parent n-grams using the first database, wherein the set of labels are in the first language; and

in response to determining that a matching threshold of a label of the set of labels is met, assigning the label to the intermediary n-gram, wherein the matching threshold is a frequency the label occurs in the set of labels associated with the parent n-grams, wherein assigning the label comprises storing the label in the first language in a second database, and wherein the second database stores the title in the second language and the intermediary n-gram in the first language.

20. A computer program product for generating a database of labeled foreign titles, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

receiving a title in a second language;

storing n-grams each with associated labels in a first language in a first database, wherein the first language and the second language are different;

sanitizing, using a processor, the title in the second language into a sanitized title in the second language;

translating the sanitized title in the second language into a translated title in the first language;

breaking the translated title in the first language into intermediary n-grams in the first language;

determining parent n-grams of an intermediary n-gram of the intermediary n-grams, wherein the intermediary n-gram is a sub-string of the parent n-grams;

retrieving a set of labels associated with the parent n-grams using the first database, wherein the set of labels are in the first language; and

in response to determining that a matching threshold of a label of the set of labels is met, assigning the label to the intermediary n-gram, wherein the matching threshold is a frequency the label occurs in the set of labels associated with the parent n-grams, wherein assigning the label comprises storing the label in the first language in a second database, and wherein the second database stores the title in the second language and the intermediary n-gram in the first language.