Optimization of text-based training set selection for language processing modules

• US 20060074924A1
• Filed: 09/17/2004
• Published: 04/06/2006
• Est. Priority Date: 09/17/2004
• Status: Active Grant

• Current Assignee: Conversant Wireless Licensing S.à.r.l. (f/k/a Core Wireless Licensing S.a.r.l.) (Conversant Intellectual Property Management, Inc. (f/k/a MOSAID Technologies))
• Original Assignee: Nokia Corporation
• Inventors: Nurminen, Jani K., Tilei, Jian
• Granted Patent: US 7,831,549 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G10L 13/08 Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
• CPC Class Codes: G10L 15/063 Training

Abstract

A device and a method provide for selection of a database from a corpus using an, optimization function. The method includes defining a size of a database, calculating a distance using a distance function for each pair in a set of pairs, and executing an optimization function using the distance to select each entry saved in the database until the number of saved entries equals the size of the database. Each pair in the set of pairs includes either two entries selected from a corpus or one entry selected from a set of previously selected entries and another entry selected from a set of a remaining portion of the corpus. The distance function may be a Levenshtein distance function or a generalized Levenshtein distance function.

Claims

1. A method of selecting a database from a corpus, the method comprising:

defining a size of a database;

calculating a coefficient for at least one pair in a set of pairs; and

executing a function to select each entry to be saved in the database until a number of entries of the database equals the size of the database.

2. The method of claim 1, wherein the coefficient is the distance between the pair.

3. The method of claim 1, wherein the function comprises an optimization function utilizing the coefficient.

4. The method of claim 1, wherein the pair comprises two entries selected from a corpus.

5. The method of claim 1, wherein the pair comprises one entry selected from a set of previously selected entries and another entry selected from a set from a remaining portion of the corpus.

6. The method of claim 1, wherein executing the function comprises:

(a) selecting an initial pair of entries from the set of pairs, wherein the distance of the initial pair is greater than or equal to the distance calculated for each pair in the set of pairs;

(b) moving the initial pair into the database;

(c) identifying a new entry from the corpus, for which the average coefficient to entries in the training database is greater than or equal to average distances calculated for all other entries in the corpus

(d) moving the new entry from the corpus into the database; and

(e) if a number of entries of the database is less than the defined size of the database, repeating (c) and (d).

7. The method of claim 1, wherein the distance function is selected from the group consisting of a Levenshtein distance function, and a generalized Levenshtein distance function.

8. The method of claim 1, wherein the set of pairs does not include the combination wherein the two entries are equal.

9. The method of claim 1, wherein the corpus contains data wherein the data is selected from the group consisting of a word, a phoneme, a grapheme, a syllable, and an onset-nucleus-coda representation.

10. A computer program product for selecting a database from a corpus, the computer code configured to:

define a size of a database;

calculate a coefficient for at least one pair in a set of pairs; and

execute a function to select each entry to be saved in the database until a number of entries of the database equals the size of the database.

11. The computer program product of claim 10, wherein the coefficient is the distance between the pair.

12. The computer program product of claim 10, wherein the function comprises an optimization function utilizing the coefficient.

13. The computer program product of claim 10, wherein the pair comprises two entries selected from a corpus.

14. The computer program product of claim 10, wherein the pair comprises one entry selected from a set of previously selected entries and another entry selected from a set from a remaining portion of the corpus.

15. The computer program product of claim 10, wherein executing the program comprises the steps of:

(a) select an initial pair of entries from the set of pairs, wherein the distance of the initial pair is greater than or equal to the distance calculated for each pair in the set of pairs;

(b) move the initial pair into the database;

(c) identify a new entry from the corpus, for which the average coefficient to entries in the training database is greater than or equal to average distances calculated for all other entries in the corpus

(d) move the new entry from the corpus into the database; and

(e) if a number of entries of the database is less than the defined size of the database, repeating (c) and (d).

16. The computer program product of claim 10, wherein the distance function is selected from the group consisting of a Levenshtein distance function, and a generalized Levenshtein distance function.

17. A device configured for selecting a database from a corpus, the device comprising a processor and computer code stored into memory, the computer code configured to:

define a size of a database;

calculate a coefficient for at least one pair in a set of pairs; and

execute a function to select each entry to be saved in the database until a number of entries of the database equals the size of the database.

18. The device of claim 17, wherein the coefficient is the distance between the pair.

19. The device of claim 17, wherein the function comprises an optimization function utilizing the coefficient.

20. The device of claim 17, wherein the pair comprises two entries selected from a corpus.

21. The device of claim 17, wherein the pair comprises one entry selected from a set of previously selected entries and another entry selected from a set from a remaining portion of the corpus.

22. The device of claim 17, wherein executing the function comprises:

(a) selecting an initial pair of entries from the set of pairs, wherein the distance of the initial pair is greater than or equal to the distance calculated for each pair in the set of pairs;

(b) moving the initial pair into the database;

(c) identifying a new entry from the corpus, for which the average coefficient to entries in the training database is greater than or equal to average distances calculated for all other entries in the corpus

(d) moving the new entry from the corpus into the database; and

(e) if a number of entries of the database is less than the defined size of the database, repeating (c) and (d).

23. A system for processing linguistic inputs to determine an output, the system comprising:

defining a size of a database;

calculating a coefficient for at least one pair in a set of pairs; and

executing a function to select each entry to be saved in the database until a number of entries of the database equals the size of the database.

24. The system of claim 23, wherein the coefficient is the distance between the pair.

25. The system of claim 23, wherein the function comprises an optimization function utilizing the coefficient.

26. The system of claim 23, wherein the pair comprises two entries selected from a corpus.

27. The system of claim 23, wherein the pair comprises one entry selected from a set of previously selected entries and another entry selected from a set from a remaining portion of the corpus.

28. A module configured for selecting a database from a corpus, the module configured to:

define a size of a database;

calculate a coefficient for at least one pair in a set of pairs; and

execute a function to select each entry to be saved in the database until a number of entries of the database equals the size of the database.

29. The module of claim 28, wherein the coefficient is the distance between the pair.

30. The module of claim 28, wherein the function comprises an optimization function utilizing the coefficient.

31. The module of claim 28, wherein the pair comprises two entries selected from a corpus.

32. The module of claim 28, wherein the pair comprises one entry selected from a set of previously selected entries and another entry selected from a set from a remaining portion of the corpus.