Real-time video segmentation on a GPU for scene and take indexing

• US 8,358,381 B1
• Filed: 04/10/2007
• Issued: 01/22/2013
• Est. Priority Date: 04/10/2007
• Status: Active Grant

• Current Assignee: NVIDIA Corporation
• Original Assignee: NVIDIA Corporation
• Inventors: Diard, Franck R.
• Primary Examiner(s): DIEP, NHON THANH
• Application Number:
  US11/733,452
• Time in Patent Office: 2,114 Days
• Field of Search: None
• US Class Current: 348/701
• CPC Class Codes: G11B 2220/2562 DVDs [digital versatile discs]; Digital video discs; MMCDs; HDCDs
• CPC Class Codes: G11B 27/005 Reproducing at a different information rate from the information rate of recording for television signals H04N5/783
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: H04N 21/4325 by playing back content from the storage medium reproduction of recorded television signals H04N5/76; reproduction of recorded television signals H04N9/79
• CPC Class Codes: H04N 21/47217 for controlling playback functions for recorded or on-demand content, e.g. using progress bars, mode or play-point indicators or bookmarks specific graphical features in visual interfaces H04N21/4312
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 5/147 Scene change detection
• CPC Class Codes: H04N 5/783 Adaptations for reproducing at a rate different from the recording rate
• CPC Class Codes: H04N 9/8205 involving the multiplexing of an additional signal and the colour video signal

Abstract

One embodiment of the present invention sets forth a technique that enables a user to reverse through video content based on scene transitions. By employing a graphics processing unit to compute one or more frame-to-frame correlation coefficients that measure the consistency of sequential images and a central processing unit to analyze the one or more correlation coefficients, a list of scene transitions may be generated in real-time. The list of scene transitions forms the basis for a content-based reverse user control within the playback application. The content-based reverse user control enables a more natural mechanism for reversing through video content, providing the user with a superior overall viewing experience.

Claims

1. A system for generating a list of content-based scene transitions within a stream of video data, the system comprising:

a video decoder engine configured to receive the video data and to generate a plurality of video frames based on the video data;

a frame buffer for storing the plurality of video frames;

a frame correlation engine configured to generate at least one correlation coefficient based on a comparison between at least two video frames in the plurality of video frames, wherein the at least one correlation coefficient is used to determine whether a scene boundary exists between any of the at least two video frames; and

a data transport engine configured to transfer the at least one correlation coefficient to a playback application executing on a central processing unit (CPU) to allow the playback application to determine whether a scene boundary exists between any of the at least two video frames based on the at least one correlation coefficient and to generate an entry in a list of scene transitions when a determination is made that a scene boundary exists.

2. The system of claim 1, wherein the frame correlation engine includes a pixel difference engine that applies a pixel difference function between each pixel in a first video frame of the at least two video frames and a corresponding pixel in a second video frame of the at least two video frames to generate a set of difference pixel values.

3. The system of claim 2, wherein the pixel difference function is applied using the luminance value associated with each pixel.

4. The system of claim 2, wherein the frame correlation engine further includes a pixel difference summing engine that adds together the difference pixel values to produce the at least one correlation coefficient.

5. The system of claim 1, wherein the frame correlation engine includes a frame histogram engine configured to generate a first histogram of the luminance values associated with the pixels in a first video frame of the at least two video frames and a second histogram of the luminance values associated with the pixels in a second video frame of the at least two video frames.

6. The system of claim 5, wherein the frame correlation engine further includes a histogram difference engine that applies a histogram difference function between an element in the first histogram and a corresponding element in the second histogram to generate a set of histogram difference values, and a histogram difference summing engine that adds together the histogram difference values to generate the at least one correlation coefficient.

7. The system of claim 1, wherein the frame correlation engine includes a pixel luminance correlation engine configured to:

apply a pixel difference function between the luminance value of each pixel in a first video frame of the at least two video frames and the luminance value of a corresponding pixel in a second video frame of the at least two video frames to generate a set of luminance difference values; and

add together the luminance difference values to produce a first correlation coefficient.

8. The system of claim 7, wherein the frame correlation engine further includes a luminance histogram correlation engine configured to:

generate a first histogram of the luminance values associated with the pixels in the first video frame and a second histogram of the luminance values associated with the pixels in the second video frame;

apply a histogram difference function between an element in the first histogram and a corresponding element in the second histogram to generate a set of histogram difference values; and

add together the histogram difference values to generate a second correlation coefficient.

9. The system of claim 1, wherein the entry indicates the time at which the scene boundary is located within the video data.

10. The system of claim 9, wherein the playback application is configured to receive user input to view the video data in a reverse mode that allows the user to view the video data beginning at the scene boundary.

11. The system of claim 10, further comprising a display engine configured to generate a video output signal to display the video data beginning at the scene boundary based on a playback time control signal generated by the playback application in response to the user input.

12. A computing device configured to generate a list of content-based scene transitions within a stream of video data, the computing device comprising:

a graphics processing unit (GPU) configured to;

receive the video data and generate a plurality of video frames based on the video data;

generate at least one correlation coefficient based on a comparison between at least two video frames in the plurality of video frames, wherein the at least one correlation coefficient is used to determine whether a scene boundary exists between any of the at least two video frames; and

transfer the at least one correlation coefficient to a playback application executing on a central processing unit (CPU); and

the playback application configured to;

determine whether a scene boundary exists between any of the at least two video frames based on the at least one correlation coefficient; and

generate an entry in a list of scene transitions when a determination is made that a scene boundary exists.

13. The computing device of claim 12, wherein the playback application is configured to receive user input to view the video data in a reverse mode and to generate a playback time control signal in response to the user input that causes the GPU to display the video data beginning at the scene boundary.

14. The computing device of claim 12, wherein the GPU computes the at least one correlation coefficient using a pixel difference function.

15. The computing device of claim 12, wherein the GPU computes the at least one correlation coefficient using a histogram difference function.

16. A method for generating a list of content-based scene transitions within a stream of video data, the method comprising:

receiving the video data and generating a plurality of video frames based on the video data;

generating at least one correlation coefficient based on a comparison between at least two video frames in the plurality of video frames, wherein the at least one correlation coefficient is used to determine whether a scene boundary exists between any of the at least two video frames; and

transferring the at least one correlation coefficient to a playback application executing on a central processing unit (CPU) to allow the playback application to determine whether a scene boundary exists between any of the at least two video frames based on the at least one correlation coefficient and to generate an entry in a list of scene transitions when a determination is made that a scene boundary exists.

17. The method of claim 16, wherein the at least one correlation coefficient is generated using a pixel difference function or a histogram difference function.