Program endpoint time detection apparatus and method, and program information retrieval system

• US 9,009,054 B2
• Filed: 10/28/2010
• Issued: 04/14/2015
• Est. Priority Date: 10/30/2009
• Status: Expired due to Fees

• Current Assignee: Institute Of Acoustics Chinese Academy Of Sciences, Sony Corporation (Sony Group Corp.)
• Original Assignee: Institute Of Acoustics Chinese Academy Of Sciences, Sony Corporation (Sony Group Corp.)
• Inventors: Liu, Kun, Wu, Weiguo, Lu, Li, Zhao, Qingwei, Yan, Yonghong, Suo, Hongbin
• Primary Examiner(s): Neway, Samuel G
• Application Number:
  US12/914,346
• Publication Number:
  US 20110106531A1
• Time in Patent Office: 1,629 Days
• Field of Search: 704/248, 704/253, 704200-218, 704270-278
• US Class Current: 704/270
• CPC Class Codes: G06F 16/68 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content

Abstract

This invention relates to retrieval for multimedia content, and provides a program endpoint time detection apparatus for detecting an endpoint time of a program by performing processing on audio signals of said program, comprising an audio classification unit for classifying said audio signals into a speech signal portion and a non-speech signal portion; a keyword retrieval unit for retrieving, as a candidate endpoint keyword, an endpoint keyword indicating start or end of the program from said speech signal portion; a content analysis unit for performing content analysis on context of the candidate endpoint keyword retrieved by the keyword retrieval unit to determine whether the candidate endpoint keyword is a valid endpoint keyword; and a program endpoint time determination unit for performing statistics analysis based on the retrieval result of said keyword retrieval unit and the determination result of said content analysis unit, and determining the endpoint time of the program. In addition, this invention also provides a program information retrieval system. With present invention, program information regarding a program attended by user can be rapidly obtained.

Claims

1. A program endpoint time detection apparatus for detecting an endpoint time of a program by performing processing on audio signals of said program, comprising:

circuitry configured to;

classify said audio signals into a speech signal portion and a non-speech signal portion; and

as to the non-speech signal portion, the circuitry is configured to;

retrieve, from the non-speech signal portion, a feature audio portion indicating a start or end of the program, and a fixed audio portion indicating the program that is about to start, and

as to the speech signal portion, the circuitry is configured to;

retrieve, as a candidate endpoint keyword, an endpoint keyword indicating start or end of the program from said speech signal portion, the endpoint keyword being retrieved only from the speech signal portion which is adjacent to the start and end of the program within a time range determined by the feature audio portion;

perform a first content analysis on a context of the retrieved candidate endpoint keyword to determine whether the retrieved candidate endpoint keyword is a valid endpoint keyword; and

perform a second content analysis on the speech signal portion to determine an expression similar to the endpoint keyword indicating start or end of the program,

wherein the circuitry is configured to perform statistics analysis based on the retrieved endpoint keyword, determination results of the first and second content analysis, the feature audio portion, and the fixed audio portion, and determine the endpoint time of the program.

2. The program endpoint time detection apparatus according to claim 1, wherein the circuitry divides the audio signals into a plurality of scene portions containing a program-on-the-spot audio portion, wherein the circuitry only classifies the program-on-the-spot audio portion of the audio signals into the speech signal portion and the non-speech signal portion.

3. The program endpoint time detection apparatus according to claim 2, wherein the circuitry only retrieves the feature audio portion from the non-speech signal portion which is adjacent to a start and end of the program-on-the-spot audio portion.

4. The program endpoint time detection apparatus according to claim 1, wherein the circuitry only retrieves the endpoint keyword from the non-speech signal portion which is adjacent to the start and end of the program within a time range determined by the fixed audio portion.

5. The program endpoint time detection apparatus according to claim 1, wherein the circuitry divides the audio signals into a plurality of scene portions containing a program-on-the-spot audio portion, wherein the circuitry only classifies the program-on-the-spot audio portion of the audio signals into the speech signal portion and the non-speech signal portion, and

the circuitry further retrieves said fixed audio portion from said plurality of scene portions.

6. The program endpoint time detection apparatus according to claim 2, wherein the circuitry detects, from the audio signals, a silence portion indicating a switching between different program portions, so as to divide the audio signals into a plurality of portions, wherein the circuitry performs an analysis on the plurality of portions, so as to be divided into said plurality of scene portions.

7. The program endpoint time detection apparatus according to claim 1, wherein the circuitry divides the audio signals into a plurality of scene portions including a program-on-the-spot audio portion, wherein the circuitry only classifies said on-the-spot audio portion of the audio signals into speech signal portion and non-speech signal portion.

8. The program endpoint time detection apparatus according to claim 1, wherein the circuitry retrieves a keyword indicating a type of the program from said speech signal portion, so as to determine the type of the program, wherein the circuitry retrieves a corresponding endpoint keyword in accordance with the type of the program.

9. The program endpoint time detection apparatus according to claim 1, wherein the circuitry performs a speech recognition on said speech signal portion and constructs a corresponding retrieval document;

and

further comprising a storage for storing a keyword indicating a program feature, said keyword including at least the endpoint keyword indicating start or end of the program,

wherein the circuitry retrieves, as the candidate endpoint keyword, the endpoint keyword indicating start or end of the program from said retrieval document, and the circuitry performs the content analysis on the context of the candidate endpoint keyword in the retrieval document, so as to determine the valid endpoint keyword.

24. The program endpoint time detection apparatus according to claim 1, wherein the feature audio portion refers to a sound specific to the program.

25. The program endpoint time detection apparatus according to claim 1, wherein the sound appears only in the program.

10. A program endpoint time detection method for detecting an endpoint time of a program by performing processing on audio signals of said program, comprising:

classifying said audio signals into a speech signal portion and a non-speech signal portion; and

as to the non-speech signal portion,

retrieving, from the non-speech signal portion, a feature audio portion indicating a start or end of the program, and a fixed audio portion indicating the program that is about to start, and

as to the speech signal portion,

retrieving, as a candidate endpoint keyword, an endpoint keyword indicating start or end of the program from said speech signal portion, the endpoint keyword being retrieved only from the speech signal portion which is adjacent to the start and end of the program within a time range determined by the feature audio portion;

performing a first content analysis on a context of the retrieved candidate endpoint keyword to determine whether the retrieved candidate endpoint keyword is a valid endpoint keyword; and

performing a second content analysis on the speech signal portion to determine an expression similar to the endpoint keyword indicating start or end of the program, and

performing, using circuitry, statistics analysis based on the retrieval result of said keyword retrieval step, determination results of said first and second content analysis step, the feature audio portion, and the fixed audio portion, and determining the endpoint time of the program.

11. The program endpoint time detection method according to claim 10, further comprising an audio scene analysis step of dividing the audio signals into a plurality of scene portions containing a program-on-the-spot audio portion, wherein in the audio classification step, only the program-on-the-spot audio portion of the audio signals is classified into the speech signal portion and the non-speech signal portion.

12. The program endpoint time detection method according to claim 11, wherein in the feature audio retrieval step, retrieval is only performed on the non-speech signal portion which is adjacent to a start and end of the program-on-the-spot audio portion.

13. The program endpoint time detection method according to claim 10, wherein in the keyword retrieval step, retrieval is only performed on the non-speech signal portion which is adjacent the start and end of the program within a time range determined by the fixed audio portion.

14. The program endpoint time detection method according to claim 10, further comprising an audio scene analysis step of dividing the audio signals into a plurality of scene portions containing a program-on-the-spot audio portion, wherein in the audio classification step, only the program-on-the-spot audio portion of the audio signals is classified into the speech signal portion and the non-speech signal portion,

wherein in the fixed audio retrieval step, said fixed audio portion is further retrieved from said plurality of scene portions divided in the audio scene analysis step.

15. The program endpoint time detection method according to claim 11, wherein the audio scene analysis step further comprises a switching gap detection step of detecting, from the audio signals, a silence portion indicating a switching between different program portions, so as to divide the audio signals into a plurality of portions, wherein in the audio scene analysis step, an analysis is performed on the plurality of portions divided in the switching gap detection step, so as to be divided into said plurality of scene portions.

16. The program endpoint time detection method according to claim 10, further comprising an audio scene analysis step of dividing the audio signals into a plurality of scene portions including a program-on-the-spot audio portion, wherein in the audio classification step, only said on-the-spot audio portion of the audio signals is classified into speech signal portion and non-speech signal portion.

17. The program endpoint time detection method according to claim 10, further comprising a program classification step of retrieving a keyword indicating a type of the program from said speech signal portion, so as to determine the type of the program, wherein in the keyword retrieval step, a corresponding endpoint keyword is retrieved in accordance with the type of the program.

18. The program endpoint time detection method according to claim 10, further comprising a speech recognition step of performing speech recognition on said speech signal portion and constructing a corresponding retrieval document,

wherein in the keyword retrieval step, the endpoint keyword indicating start or end of the program is extracted from a storage unit, and the endpoint keyword indicating start or end of the program is retrieved as the candidate endpoint keyword from said retrieval document, and in the content analysis step, a content analysis is performed on the context of the candidate endpoint keyword in the retrieval document, so as to determine the valid endpoint keyword.

19. A program information retrieval system for obtaining program information desired by a user by performing processing on audio signals of a program, comprising:

a display for displaying a user interface of said system;

a program import apparatus for importing a plurality of programs into said program information retrieval system and displaying a program list on said display apparatus; and

a program endpoint time detection apparatus for analyzing a program selected by the user from said program list, so as to obtain the endpoint time of said program and display the endpoint time of said program on said display apparatus, wherein said program endpoint time detection apparatus comprises;

circuitry configured to

classify said audio signals into a speech signal portion and a non-speech signal portion; and

as to the non-speech signal portion, the circuitry is configured to;

retrieve, from the non-speech signal portion, a feature audio portion indicating a start or end of the program, and a fixed audio portion indicating the program that is about to start, and

as to the speech signal portion, the circuitry is configured to;

retrieve, as a candidate endpoint keyword, an endpoint keyword indicating start or end of the program from said speech signal portion, the endpoint keyword being retrieved only from the speech signal portion which is adjacent to the start and end of the program within a time range determined by the feature audio portion;

perform a first content analysis on a context of the retrieved candidate endpoint keyword to determine whether the retrieved candidate endpoint keyword is a valid endpoint keyword; and

perform a second content analysis on the speech signal portion to determine an expression similar to the endpoint keyword indicating start or end of the program,

wherein the circuitry is configured to perform statistics analysis based on the retrieved endpoint keyword, determination results of the first and second content analysis, the feature audio portion, and the fixed audio portion, and determine the endpoint time of the program.

20. The program information retrieval system according to claim 19, wherein the circuitry classifies said plurality of programs imported by the program import apparatus in accordance with a category to which the program belongs, and the circuitry controls the display to display, according to a category instructed by the user, a program list of this category.

21. The program information retrieval system according to claim 19, wherein the circuitry retrieves a cheering clip occurred in the program, and the circuitry controls the display to display time information thereof.

22. The program information retrieval system according to claim 19, wherein the circuitry retrieves a portion in the program in which a specific speaker is speaking, and the circuitry controls the display to differently display time information about the portion spoken by different speakers.

23. The program information retrieval system according to claim 19, wherein the circuitry controls the display to display a keyword list corresponding to the program selected by the user from the program list and retrieves corresponding keyword according to a selection of the user, and controls the display to display time information about the keyword.