Lattice matching

• US 7,590,605 B2
• Filed: 07/16/2004
• Issued: 09/15/2009
• Est. Priority Date: 07/16/2003
• Status: Expired due to Fees

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Josifovski, Ljubomir
• Primary Examiner(s): Vincent; David R
• Assistant Examiner(s): Brown, Jr.; Nathan H
• Application Number:
  US10/564,632
• Publication Number:
  US 20070038450A1
• Time in Patent Office: 1,887 Days
• Field of Search: 706/29, 704/236, 707/3, 707/6
• US Class Current: 706/29
• CPC Class Codes: G06F 16/3343 using phonetics
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/025 Phonemes, fenemes or fenones being the recognition units
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99936 Pattern matching access

Abstract

A system is described for matching lattices such as phoneme lattices generated by an automatic speech recognition unit. The system can be used to retrieve files from a database by comparing a query lattice with annotation lattices associated with the data files that can be retrieved, and by retrieving the data files having an annotation lattice most similar to the query lattice.

Claims

1. A method of searching a database comprising a plurality of information entries to identify stored image data to be retrieved therefrom, each of said plurality of information entries having an associated annotation lattice, the method comprising:

using a user terminal to receive a query lattice representing an input query;

comparing the query lattice with each annotation lattice to provide a set of comparison results;

identifying stored image data to be retrieved from said database using the set of comparison results; and

displaying said stored image data retrieved from said database on a display of the user terminal;

wherein the query lattice and each annotation lattice define alternative label sequences that represent a sequential signal, and each lattice comprises a plurality of nodes each associated with one or more labels and representing a point in the sequential signal at which the associated label occurs,

wherein the query lattice is compared with each annotation lattice by propagating a plurality of paths, each path representing a comparison between labels in the query lattice and labels in the annotation lattice, and each path having an associated accumulative value representing the closeness of the comparison,

wherein during the path propagation, said comparing step defines, for each node in the query lattice, a plurality of associated storage areas in a memory, each storage area associated with a query lattice node also being associated with a respective node in the annotation lattice and being operable to store, during the path propagation, an accumulative value representing the closeness of the comparison between labels in the query lattice up to the associated query lattice node and labels in the annotation lattice up to the associated annotation lattice node,

wherein said comparing step uses said storage areas during the propagation of said paths, and

wherein when propagating a path from a source node in said query lattice to a destination node in said query lattice, said comparing step updates and propagates accumulative values stored in the storage areas associated with the source node to at least the storage areas associated with the destination node.

2. A method according to claim 1, wherein during the propagation of said accumulative values, said comparing step compares the appropriate accumulative value in the storage area associated with the destination node with the updated accumulative value from the storage area associated with the source node.

3. A method according to claim 1, wherein said comparing step updates the accumulative values stored in the storage areas associated with the source node to take into account for substitution of the corresponding labels in the query lattice and the corresponding labels in the annotation lattice.

4. A method according to claim 1, wherein said comparing step updates the accumulative value stored in the storage areas associated with the source node to take into account the substitution, insertion and deletion of labels from the query lattice and/or annotation lattice and wherein the storage area to which an updated score is propagated depends upon whether a label is substituted, inserted or deleted from the query lattice and/or the annotation lattice.

5. A method according to claim 1, wherein said comparing step updates the accumulative values stored in the storage areas associated with the source node by comparing the corresponding labels in the query lattice with the corresponding labels in the annotation lattice.

6. A method according to claim 1, wherein said query lattice is generated by a recognition unit and includes confidence data associated with said labels indicative of the confidence that said recognition unit correctly recognized the label and wherein said comparing step updates said accumulative values stored in the storage areas associated with the source node using the confidence data from the corresponding labels.

7. A method according to claim 1, wherein the storage areas associated with nodes in the query lattice are stored in a plurality of node tables, and each of the storage areas associated with a node in the query lattice are stored in a node table associated with the respective query lattice node.

8. A method according to claim 2, wherein said comparing step replaces the accumulative value in the storage area associated with the destination node with the updated accumulative value from the storage area associated with the source node if the updated accumulative value is better than the accumulative value stored in the storage area associated with the destination node.

9. A method according to claim 5, wherein said comparing step updates said accumulative values by using predetermined confusion data which defines measures of confusability between the different labels.

10. A method according to claim 6, wherein both said query and annotation lattices include said confidence data and wherein said comparing step updates said accumulative values using the confidence data for the respective labels from the query and annotation lattices.

31. A method according to claim 7, wherein said storage areas in said node tables are arranged in a sequential order defined by the sequential order of the associated nodes.

11. A method of searching a database comprising a plurality of information entries to identify stored image data to be retrieved therefrom, each of said plurality of information entries having an associated annotation lattice, the method comprising:

using a user terminal to receive a query lattice representing an input query;

comparing the query lattice with each annotation lattice to provide a set of comparison results;

identifying stored image data to be retrieved from said database using the set of comparison results; and

displaying said stored image data retrieved from said database on a display of the user terminal;

wherein the query lattice and each annotation lattice define alternative label sequences that represent a sequential signal, and each lattice comprises a plurality of nodes each associated with one or more labels and representing a point in the sequential signal at which the associated label occurs,

wherein the query lattice is compared with each annotation lattice by propagating a plurality of paths, each path representing a comparison between labels in the query lattice and labels in the annotation lattice, and each path having an associated accumulative value representing the closeness of the comparison,

wherein during the path propagation, said comparing step defines, for each node in the query lattice, a plurality of associated storage areas in a memory, each storage area associated with a query lattice node also being associated with a respective node in the annotation lattice and being operable to store, during the path propagation, an accumulative value representing the closeness of the comparison between labels in the query lattice up to the associated query lattice node and labels in the annotation lattice up to the associated annotation lattice node,

wherein said comparing step uses said storage areas during the propagation of said paths,

wherein the method further comprises a processing step of processing accumulative values stored for a node to determine a similarity measure representing the similarity between the query and annotation lattices, and

wherein the annotation lattice represents a longer sequence than the query lattice and wherein the processing step processes the accumulative values stored for the node to determine if the annotation lattice includes one or more portions similar to the query lattice.

12. A method according to claim 11, wherein said processing step compares the accumulative values in the storage areas of the node to identify values better than a predetermined threshold, to identify said one or more portions in the annotation lattice which are similar to the fist lattice.

13. A method according to claim 12, wherein said processing step identifies said one or more portions by identifying the storage areas having an accumulative value better than said threshold.

14. A method according to claim 12 or 13, wherein the sequence length of the query lattice is known, wherein when said processing step identifies an accumulative value better than said threshold, the annotation lattice node associated with the identified accumulative value represents the end of said portion corresponding to said query lattice and wherein said processing step estimates a beginning of the portion within the annotation lattice using the known sequence length of the query lattice.

15. An apparatus for searching a database comprising a plurality of information entries to identify stored image data to be retrieved therefrom, each of said plurality of information entries having an associated annotation lattice, the apparatus comprising:

a memory;

a receiver operable to receive a query lattice representing an input query;

a lattice comparison apparatus for comparing the query lattice with each annotation lattice to provide a set of comparison results;

an identifier operable to identify stored image data to be retrieved from said database using the set of comparison results provided by the lattice comparison apparatus; and

a display operable to display said stored image data identified by the identifier and retrieved from said database;

wherein the query lattice and each annotation lattice define alternative label sequences that represent a sequential signal, and each lattice comprises a plurality of nodes each associated with one or more labels and representing a point in the sequential signal at which the associated label occurs,

wherein the query lattice is compared with each annotation lattice by propagating a plurality of paths, each path representing a comparison between labels in the query lattice and labels in the annotation lattice, and each path having an associated accumulative value representing the closeness of the comparison,

wherein during the path propagation, said lattice comparison apparatus is operable to define, for each node in the query lattice, a plurality of associated storage areas in said memory, each storage area associated with a query lattice node also being associated with a respective node in the annotation lattice and being operable to store, during the path propagation, an accumulative value representing the closeness of the comparison between labels in the query lattice up to the associated query lattice node and labels in the annotation lattice up to the associated annotation lattice node,

wherein said lattice comparison apparatus is operable to use said storage areas during the propagation of said paths,

wherein when propagating a path from a source node in said query lattice to a destination node in said query lattice, said lattice comparison apparatus is operable to update and to propagate accumulative values stored in the storage areas associated with the source node to at least the storage areas associated with the destination node.

16. An apparatus according to claim 15, wherein during the propagation of said accumulative values, said lattice comparison apparatus is operable to compare the appropriate accumulative value in the storage area associated with the destination node with the updated accumulative value from the storage area associated with the source node.

17. An apparatus according to claim 15, wherein said lattice comparison apparatus is operable to update the accumulative values stored in the storage areas associated with the source node to take into account for substitution of the corresponding labels in the query lattice and the corresponding labels in the annotation lattice.

18. An apparatus according to claim 15, wherein said lattice comparison apparatus is operable to update the accumulative value stored in the storage areas associated with the source node to take into account the substitution, insertion and deletion of labels from the query lattice and/or annotation lattice and wherein the storage area to which an updated score is propagated depends upon whether a label is substituted, inserted or deleted from the query lattice and/or the annotation lattice.

19. An apparatus according to claim 15, wherein said lattice comparison apparatus is operable to update the accumulative values stored in the storage areas associated with the source node by comparing the corresponding labels in the query lattice with the corresponding labels in the annotation lattice.

20. An apparatus according to claim 15, wherein said query lattice is generated by a recognition unit and includes confidence data associated with said labels indicative of the confidence that said recognition unit correctly recognized the label and wherein said lattice comparison apparatus is operable to update said accumulative values stored in the storage areas associated with the source node using the confidence data for the corresponding labels.

21. An apparatus according to claim 15, wherein the storage areas associated with nodes in the query lattice are stored in a plurality of node tables, and each of the storage areas associated with a node in the query lattice are stored in a node table associated with the respective query lattice node.

22. An apparatus according to claim 15, wherein said set of comparison results includes at least one score representing the similarity between the query lattice and each annotation lattice and wherein said identifier is operable to identify the N most relevant information entries by ranking the scores within said set of comparison results to identify the N-best scores.

23. An apparatus according to claim 16, wherein said lattice comparison apparatus is operable to replace the accumulative value in the storage area associated with the destination node with the updated accumulative value from the storage area associated with the source node if the updated accumulative value is better than the accumulative value stored in the storage area associated with the destination node.

24. An apparatus according to claim 19, wherein said lattice comparison apparatus is operable to update said accumulative values by using predetermined confusion data which defines measures of confusability between the different labels.

25. An apparatus according to claim 20, wherein both said query and annotation lattices include said confidence data and wherein said lattice comparison apparatus is operable to update said accumulative values using the confidence data for the respective labels from the query and annotation lattices.

26. An apparatus according to claim 21, wherein said storage areas in said node tables are arranged in a sequential order defined by the sequential order of the associated nodes.

27. An apparatus for searching a database comprising a plurality of information entries to identify stored image data to be retrieved therefrom, each of said plurality of information entries having an associated annotation lattice, the apparatus comprising:

a memory;

a receiver operable to receive a query lattice representing an input query;

a lattice comparison apparatus for comparing the query lattice with each annotation lattice to provide a set of comparison results;

an identifier operable to identify stored image data to be retrieved from said database using the set of comparison results provided by the lattice comparison apparatus; and

a display operable to display said stored image data identified by the identifier and retrieved from said database;

wherein the query lattice and each annotation lattice define alternative label sequences that represent a sequential signal, and each lattice comprises a plurality nodes each associated with one or more labels and representing a point in the sequential signal at which the associated label occurs,

wherein the query lattice is compared with each annotation lattice by propagating a plurality of paths, each path representing a comparison between labels in the query lattice and labels in the annotation lattice, and each path having an associated accumulative value representing the closeness of the comparison,

wherein during the path propagation, said lattice comparison apparatus is operable to define, for each node in the query lattice, a plurality of associated storage areas in said memory, each storage area associated with a query lattice node also being associated with a respective node in the annotation lattice and being operable to store, during the path propagation, an accumulative value representing the closeness of the comparison between labels in the query lattice up to the associated query lattice node and labels in the annotation lattice up to the associated annotation lattice node,

wherein said lattice comparison apparatus is operable to use said storage areas during the propagation of said paths,

wherein the apparatus further comprises a processor operable to process the accumulative values stored for a node to determine a similarity measure representing the similarity between the query and annotation lattices, and

wherein the annotation lattice represents a longer sequence than the query lattice and wherein the processor is operable to process the accumulative values stored for the node to determine if the annotation lattice includes one or more portions similar to the query lattice.

28. An apparatus according to claim 27, wherein said processor is operable to compare the accumulative values in the storage areas of the node to identify values better than a predetermined threshold, to identify said one or more portions in the annotation lattice which are similar to the query lattice.

29. An apparatus according to claim 28, wherein said processor is operable to identify said one or more portions by identifying the storage areas having an accumulative value better than said threshold.

30. An apparatus according to claim 28, wherein the sequence length of the query lattice is known, wherein when said processor identifies an accumulative value better than said threshold, the annotation lattice node associated with the identified accumulative value represents the end of the said portion corresponding to said query lattice and wherein said processor is operable to estimate a beginning of the portion within the annotation lattice using the known sequence length of the query lattice.