METHOD FOR INDEXING MULTIMEDIA INFORMATION

• US 20110320197A1
• Filed: 04/27/2011
• Published: 12/29/2011
• Est. Priority Date: 06/23/2010
• Status: Active Grant

• Current Assignee: Telefonica SA
• Original Assignee: Telefonica SA
• Inventors: CONEJERO, David, DUXANS, Helenca, ESCALADA, Gregorio
• Granted Patent: US 8,775,174 B2
• Time in Patent Office: Days
• US Class Current: 704/235
• CPC Class Codes: G06F 16/41 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/433 using audio data
• CPC Class Codes: G06F 16/61 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/685 using automatically derived transcript of audio data, e.g. lyrics speech recognition G10L15/00
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

It comprises analyzing audio content of multimedia files and performing a speech to text transcription thereof automatically by means of an ASR process, and selecting acoustic and language models adapted for the ASR process at least before the latter processes the multimedia file, i.e. “a priori”.

The method is particularly applicable to the automatic indexing, aggregation and clustering of news from different sources and from different types of files, including text, audio and audiovisual documents without any manual annotation.

Claims

1. A method for indexing multimedia information, comprising analyzing audio content of multimedia files and performing a speech to text transcription thereof, wherein said speech to text transcription is performed automatically, for each multimedia file, by means of an Automatic Speech Recognition, or ASR, process, and in that the method comprises selecting acoustic and language models adapted for said ASR process at least before the latter processes the multimedia file.

2. A method as per claim 1, wherein said multimedia information also comprises text files, the method comprising automatically aggregating information from said text files to information from said multimedia files with audio content.

3. A method as per claim 2, further comprising automatically clustering said multimedia information.

4. A method as per claim 1, wherein said multimedia files with audio content comprise at least one of audio files and audiovisual files.

5. A method as per claim 1, wherein said selection of acoustic models is performed by carrying out, on each multimedia file, at least a speaker diarization.

6. A method as per claim 5, wherein said speaker diarization is carried out by performing three main operations:

a segmentation of the audio track related to said multimedia file audio content into similar acoustic segments;

an identification of the acoustic characteristics of each segment, or cluster of segments, obtained from said segmentation; and

a selection of the most adequate ASR acoustic model from an available set.

7. A method as per claim 6, wherein said segmentation is performed by removing the non-speech parts using a Voice Activity Detection and applying a diarization technique based on a Bayesian information criterion for clustering related segments.

8. A method as per claim 7, wherein said identification of acoustic characteristics is carried out by processing each cluster obtained from said clustering by an open set acoustic identifier using LLR, log-likelihood ratio, as a decision score, said identification given as a result one of a positive identification and an unknown identification.

9. A method as per claim 8, wherein said selection of the most adequate ASR acoustic model comprises:

for said positive identification, selecting a corresponding adapted ASR acoustic model associated to said acoustic characteristics, or

for said unknown identification, selecting a general ASR acoustic model for multimedia applications.

10. A method as per claim 1, wherein said selection of language models is performed by carrying out, on textual information of each multimedia file, at least a textual diarization and a subject identification.

11. A method as per claim 10, wherein said textual information comes from subtitles and/or closed caption contained in the multimedia file and/or from a previous transcription thereof.

12. A method as per claim 10, wherein said textual information includes temporal information representing its temporal position in the multimedia file audio content, said textual diarization comprising splitting the multimedia file in temporal chunks or pseudo-sentences, and wherein a subject classification is performed for each of said chunks.

13. A method as per claim 12, comprising obtaining each of said pseudo-sentences with its classification by:

performing, for each pseudo-sentence, a context aggregation with a temporal criterion;

applying a standard classification strategy to every contextual aggregated pseudo-sentence, based on a set of training categories, to obtain all class hypotheses with their scores are kept;

applying a time-weighted rank ordering filter to select the best class hypothesis for the sentence to be classified; and

performing a postprocessing in order to avoid too short sequences of one class, if it lays between two long sequences with the same class, where only “

welcome” and

“

farewell”

classes, laying respectively at the beginning and end of the indexed multimedia information, are allowed to elude this condition.

14. A method as per claim 10, comprising, once the multimedia file has been acoustically and textually diarized, selecting the acoustic and language models to perform the best recognition possible on each part of the multimedia file considering every acoustic event and every textual event as independent events, and considering that the beginning of the multimedia file is assigned to acoustic and language general models.

15. A method as per claim 14, comprising, for each new event of said independent events, if it happens after a previous event a time over a determined threshold, changing models accordingly as a result of new diarizations.

16. A method as per claim 1, comprising aggregating standard or close-caption subtitle textual information of a multimedia file to the text transcription performed by the ASR process.

17. A method as per claim 16, comprising improving said selection of said language model by iteratively diarizing a transcription of a previous cycle of ASR processing with said textual information, if existing, aggregated thereto.

18. A method as per claim 3, comprising:

lemmatizing the multimedia files text transcription and the text data of text files and

indexing said multimedia information by detecting all the relevant concepts in the multimedia and text files and keeping track of the place, for text files, or instant, for audiovisual files, of their appearances.

19. A method as per claim 18, wherein said clustering of multimedia information comprises selecting the keywords among all the relevant indexed concepts to generate a label with the relationship between all the files that contains the same keyword, and by a final selection, for each concept, of the nearest keywords.

20. A method as per claim 19, wherein said keywords selection is performed by applying a modified TFIDF algorithm to the lemmatized text to select a set of lemmatized words extracted from all the indexing files according to its relevance for a single file or for a set of files, where said modified TFID algorithm is based on:

21. A method as per claim 20, comprising performing, for each keyword, an extraction of a list of related keywords by calculating the correlation of each pair of words X and Y by means of the following formula

22. A method as per claim 1, wherein said multimedia information refers to news, the method comprising indexing said news from different sources.