Apparatus and method for retrieving a desired video/voice scenes using voice recognition

• US 20030200091A1
• Filed: 05/09/2003
• Published: 10/23/2003
• Est. Priority Date: 12/17/1998
• Status: Active Grant

• Current Assignee: Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Original Assignee: Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Inventors: Hoshimi, Masakatsu, Inoue, Ikuo, Endo, Mitsuru, Furuyama, Hiroshi, Yashio, Hitoshi
• Granted Patent: US 6,728,673 B2
• Time in Patent Office: Days
• US Class Current: 704/254
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/71 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data

Abstract

A video retrieval data generation apparatus includes an extractor that is configured to extract a characteristic pattern from a voice signal synchronous with a video signal. The video retrieval data generation apparatus also includes an index generator that is configured to set the voice signal for a voice period as a processing target. The index generator is further configured to prepare standard voice patterns of a subword corresponding to a plurality of subwords, detect, for each subword, a characteristic pattern similar to a standard voice pattern at each of the voice periods, and generate, for each subword, an index containing time synchronization information corresponding to a position where the similar characteristic pattern is detected. The video retrieval data generation apparatus also includes a multiplexer that is configured to multiplex video signals, voice signals and indexes to output in a data stream format.

Claims

1. A video retrieval data generation apparatus comprising:

an extractor that is configured to extract a characteristic pattern from a voice signal synchronous with a video signal;

an index generator that is configured to set the voice signal for a voice period as a processing target, to prepare standard voice patterns of a subword, to detect, at each voice period, for each subword, a characteristic pattern similar to a standard voice pattern, and to generate, for each subword, an index containing time synchronization information corresponding to a position where the similar characteristic pattern is detected; and

a multiplexer that is configured to multiplex video signals, voice signals and indexes to output in a data stream format.

2. The video retrieval data generation apparatus according to claim 1, wherein a data stream output from the multiplexer is provided to a video retrieval apparatus through one of a broadcast network and a communication network.

3. The video retrieval data generation apparatus according to claim 1, wherein a data stream output from the multiplexer is stored in a storage medium.

4. The video retrieval data generation apparatus according to claim 1, wherein the multiplexer divides the indexes into units that are brought into coincidence with unit GOP (Group of Pictures) of corresponding video signals.

5. The video retrieval data generation apparatus according to claim 1, wherein the multiplexer multiplexes video signals, voice signals and indexes on an MPEG stream.

6. The video retrieval data generation apparatus according to claim 1, wherein the index contains a type of the subword, a beginning time of the characteristic pattern, and a score indicative of a similarity between the standard voice pattern and the characteristic pattern.

7. A video retrieval data generation method, comprising:

extracting a characteristic pattern from a voice signal synchronous with a video signal;

setting the voice signal for a voice period as a processing target, preparing standard voice patterns of a subword, detecting, at each voice period, for each subword, a characteristic pattern similar to a standard voice pattern, and generating, for each subword, an index containing time synchronization information corresponding to a position where the similar characteristic pattern is detected; and

multiplexing video signals, voice signals and indexes to output in a data stream format.

8. The video retrieval data generation method according to claim 7, further comprising:

providing a data stream obtained in the multiplexing to a video retrieval apparatus through one of a broadcast network and a communication network.

9. The video retrieval data generation method according to claim 7, further comprising:

storing a data stream obtained in the multiplexing in a storage medium.

10. The video retrieval data generation method according to claim 7, wherein generating the index comprises providing a type of the subword, a beginning time of the characteristic pattern, and a score indicative of a similarity between the standard voice pattern and the characteristic pattern.

11. A video retrieval apparatus comprising:

a demultiplexer that is configured to demultiplex a data stream on which are multiplexed video signals, voice signals synchronous with the video signals, and indexes generated from the voice signals on a subword basis, into at least the indexes; and

a retrieval processor that is configured to obtain time information for an input keyword from a combination of the indexes to retrieve a desired video, wherein each of the indexes contains time synchronization information indicative of a position of a characteristic pattern in the voice signals, the characteristic pattern being similar to a standard voice pattern of a subword corresponding to each of the indexes.

12. The video retrieval apparatus according to claim 11, wherein the retrieval processor comprises a collator that is configured to pick up indexes corresponding to the input keyword from the demultiplexed indexes, and obtain time information from a combination of the picked up indexes, and a control section that is configured to retrieve a video corresponding the input keyword using the obtained time information.

13. The video retrieval apparatus according to claim 12, wherein the collator converts the input keyword into a subword sequence, picks up the indexes according to the order of the subword sequence, adds similarities for each subword to obtain a similarity for the input keyword, and obtains time information of a period at which the obtained similarity has a high degree.

14. The video retrieval apparatus according to claim 11, further comprising:

a storage that is configured to store the data stream on which the video signals, the voice signals and the indexes are multiplexed without demultiplexing the data stream,

wherein the desired video is retrieved from the storage.

15. The video retrieval apparatus according to claim 11, further comprising:

a storage that is configured to store the video signals and the voice signals,

wherein the demultiplexer demultiplexes the data stream into the video signals and the voice signals along with the indexes, and stores the demultiplexed video signals and voice signals in the storage, and the desired signal is retrieved from the storage.

16. The video retrieval apparatus according to claim 11, wherein the data stream is received through one of a broadcast network and a communication network.

17. The video retrieval apparatus according to claim 11, wherein the data stream is read from a storage medium.

18. The video retrieval apparatus according to claim 11, wherein the indexes are divided into units that are brought into coincidence with unit GOP (Group of Pictures) of corresponding video signals.

19. A video retrieval method, comprising:

demultiplexing a data stream on which are multiplexed video signals, voice signal synchronous with the video signals, and indexes generated from the voice signals on a subword basis, into at least the indexes; and

obtaining time information for an input keyword from a combination of the indexes to retrieve a desired video,

wherein each of the indexes contains time synchronization information indicative of a position of a characteristic pattern in the voice signals, the characteristic pattern being similar to a standard voice pattern of a subword corresponding to each of the indexes.

20. The video retrieval method according to claim 19, wherein the obtaining comprises picking up indexes corresponding to the input keyword from the demultiplexed indexes, and obtaining time information from a combination of the picked up indexes, and retrieving a video corresponding the input keyword using the obtained time information.

21. The video retrieval method according to claim 20, wherein in picking up indexes, the input keyword is converted into a subword sequence, the indexes are picked up according to the order of the subword sequence, similarities for each subword are added to obtain a similarity for the input keyword, and time information of a period at which the obtained similarity has a high degree is obtained.

22. A video recording apparatus comprising:

an extractor that is configured to extract a characteristic pattern from a voice signal synchronously input with a video signal in recording a video;

an index generator that is configured to set the voice signal for a voice period as a processing target, to prepare standard voice patterns on a subword basis, to detect, for each subword, a characteristic pattern similar to a standard voice pattern at each voice period, and to generate, for each subword, an index containing time synchronization information corresponding to a position where the similar characteristic pattern is detected;

a multiplexer that is configured to multiplex input video signals, input voice signals, and indexes, to output in a data stream format; and

a video storage medium in which a data stream output from the multiplexer is stored.

24. The video recording apparatus according to claim 22, wherein the multiplexer multiplexes the input video signals, the input voice signals and the indexes on an MPEG stream.

25. The video recording apparatus according to one of claim 22, wherein the data stream is stored in the video storage medium according to storage format of the video storage medium.

23. A video recording apparatus comprising:

an extractor that is configured to extract a characteristic pattern from a voice signal synchronously input with a video signal in recording a video;

an index generator that is configured to generate packets on a time basis while maintaining a time series of extracted characteristic patterns so as to generate indexes where each of the packets contains time information;

a multiplexer that is configured to multiplex input video signals, input voice signals and indexes to output in a data stream format; and

a video storage medium in which a data stream output from the multiplexer is stored.

26. A video reproducing apparatus comprising:

a video storage that stores a data stream on which video signals, voice signals synchronous with the video signals, and indexes generated from the voice signals on a subword basis, are multiplexed, each of the indexes containing time synchronization information indicative of a position of a characteristic pattern in the voice signals, the characteristic pattern being similar to a standard voice pattern of a subword corresponding to each of the indexes;

a read processor that is configured to read the video signals and the voice signals from the video storage while maintaining synchronization in reproducing a video, and to demultiplex the indexes from the data stream stored in the video storage when a video retrieval instruction is given;

a key word convertor that is configured to convert an input key word into time-series data on a subword basis;

a key word collator that is configured to collate the time-series data of the input key word with the indexes to obtain time information of a period at which the time series data is similar to the indexes; and

a controller that is configured to instruct the read processor to read the video signals and the voice signals using a position specified by the obtained time information as a read beginning position.

27. A video reproducing apparatus comprising:

a video storage that stores a data stream on which video signals, voice signals synchronous with the video signals, and indexes packetized on a time basis while maintaining a time series of characteristic patterns extracted from the voice signals, are multiplexed, where each packet contains time information;

a read processor that is configured to read the video signals and the voice signals from the video storage while maintaining synchronization in reproducing a video, and to demultiplex the indexes from the data stream stored in the video storage when a video retrieval instruction is given;

a key word converter that is configured to convert an input key word into time-series data of the characteristic patterns;

a key word collator that is configured to collate the time series data of the input key word with the indexes to obtain time information of a period at which the time series data is similar to the indexes; and

a controller that is configured to instruct the read processor to read the video signal and the voice signal using a position specified by the obtained time information as a read beginning position.

28. A video reproducing method, comprising:

storing, in a video storage, a data stream on which are multiplexed video signals, voice signals synchronous with the video signals, and indexes generated from the voice signals on a subword basis, each of the indexes containing time synchronization information indicative of a position of a characteristic pattern that is similar to a standard voice pattern of a subword corresponding to each of the indexes;

reading the video signals and the voice signals from the video storage while maintaining synchronization in reproducing a video, and demultiplexing the indexes from the data stream stored in the video storage when a video retrieval instruction is given;

converting an input key word into time-series data on a subword basis;

collating the time-series data of the input key word with the indexes to obtain time information of a period at which the time series data is similar to the indexes; and

reading the video signal and the voice signal using a position specified by the obtained time information as a read beginning position.

29. A video reproducing method, comprising:

storing, in a video storage, a data stream on which are multiplexed video signals, voice signals synchronous with the video signals, and indexes packetized on a time basis while maintaining a time series of characteristic patterns extracted from the voice signals, where each packet contains time information;

reading the video signals and the voice signals from the video storage while maintaining synchronization in reproducing a video, and demultiplexing the indexes from the data stream stored in the video storage when a video retrieval instruction is given;

converting an input key word into time-series data of the characteristic patterns;

collating the time series data of the input key word with the indexes to obtain time information of a period at which the time series data is similar to the indexes; and

reading the video signal and the voice signal using a position specified by the obtained time information as a read beginning position.