Multimedia data management by speech recognizer annotation

• US 7,739,110 B2
• Filed: 12/01/2006
• Issued: 06/15/2010
• Est. Priority Date: 06/07/2006
• Status: Active Grant

• Current Assignee: Industrial Technology Research Institute
• Original Assignee: Industrial Technology Research Institute
• Inventors: Lai, Yu-Sheng, Kang, Chia-Hao, Wu, Chung-Hsien, Huang, Chien-Lin
• Primary Examiner(s): Lerner; Martin
• Application Number:
  US11/565,628
• Publication Number:
  US 20070288237A1
• Time in Patent Office: 1,292 Days
• Field of Search: 704/231, 704/236, 704/239, 704/243, 704/245, 704/255, 704/270, 704/235, 707/3
• US Class Current: 704/235
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 21/06 Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids G10L15/26 takes precedence

Abstract

A method and an apparatus for multimedia data management are disclosed. The method provides an indexing and retrieval scheme for digital photos with speech annotations based on image-like patterns transformed from the recognized syllable candidates. For annotated spoken content, the recognized n-best syllable candidates are transformed into a sequence of syllable-transformed patterns. Eigen-image analysis is further adopted to extract the significant information to reduce the dimensionality. Vector quantization is applied to quantize the syllable-transformed patterns into feature vectors for indexing. The invention of indexing scheme reduces the dimensionality and noise of data, and achieves better performance of 16.26% for speech annotated photo retrieval.

Claims

1. A method for multimedia data management, comprising the steps of:

generating a recognition result for a speech-annotation feature through an automatic speech recognizer;

obtaining n-best syllable candidates of said speech-annotation feature from said recognition result, said n being a nature number;

establishing a confusion matrix for said n-best syllable candidates, said confusion matrix comprising a matrix of similarity scores, each similarity score measuring similarity of each syllable candidate against one of said n-best syllable candidates;

transforming said confusion matrix into an image with each pixel of said image having a gray scale representing the similarity measured by a corresponding similarity score;

constructing an index database by capturing eigen-images from said image using an eigen-image processing method; and

searching for said multimedia data using said index database with natural speech as input.

2. The method as claimed in claim 1, wherein said step of constructing said index database further includes:

extracting one or more feature parameters of said image; and

quantizing said feature parameters and completing index construction for said index database.

3. The method as claimed in claim 2, wherein said step of quantizing feature parameters and completing index construction further includes:

quantizing each eigen-image captured by said eigen-image processing method into a multidimensional vector-data; and

clustering the multidimensional vectors into a codeword sequence, and collecting all codeword sequences into a codebook.

4. The method as claimed in claim 1, wherein said step of searching for said multimedia data with natural speech as input further includes:

inputting speech as a query condition, and transforming said query condition into an index data;

comparing said index data to index information of said index data in said index database; and

finding corresponding multimedia data matching said query condition.

5. The method as claimed in claim 1, wherein said automatic speech recognizer is a syllable-based speech recognizer.

6. The method as claimed in claim 1, wherein said eigen-image processing method is a principal component analysis of said image.

7. An apparatus for multimedia data management, comprising:

a multimedia data index production module having;

an automatic speech recognizer for generating a recognition result from a speech-annotation feature, said recognition result including n-best syllable candidates of said speech-annotation feature, said n being a natural number;

an image simulation module for establishing a confusion matrix for said n-best syllable candidates and transforming said confusion matrix into an image, said confusion matrix comprising a matrix of similarity scores, each similarity score measuring similarity of each syllable candidate against one of said n-best syllable candidates, and each pixel of said image having a gray scale representing the similarity measured by a corresponding similarity score; and

an eigen-image capture and index construction module for constructing an index database by capturing eigen-images from said image using an eigen-image processing method; and

a multimedia data index searching module, for searching for said multimedia data using said index database with natural speech as input.

8. The apparatus as claimed in claim 7, wherein each index in said index database is a codeword sequence.

9. The apparatus as claimed in claim 7, wherein said automatic speech recognizer is a syllable-based speech recognizer.