Real-time video segmentation on a GPU for scene and take indexing

• US 8,345,769 B1
• Filed: 04/10/2007
• Issued: 01/01/2013
• Est. Priority Date: 04/10/2007
• Status: Active Grant

• Current Assignee: NVIDIA Corporation
• Original Assignee: NVIDIA Corporation
• Inventors: Diard, Franck R.
• Primary Examiner(s): Vu, Kim
• Assistant Examiner(s): BEHESHTI SHIRAZI, SAYED ARESH
• Application Number:
  US11/733,459
• Time in Patent Office: 2,093 Days
• Field of Search: 375/240.25, 375/240.18, 725/38, 455/439, 711/112
• US Class Current: 375/240.24
• CPC Class Codes: G06F 16/7847 using low-level visual features of the video content
• CPC Class Codes: G06V 10/955 using specific electronic processors
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes

Abstract

One embodiment of the present invention sets forth a technique that enables a user to reverse through video content based on scene transitions. By employing a graphics processing unit to compute one or more frame-to-frame correlation coefficients that measure the consistency of sequential images and a central processing unit to analyze the one or more correlation coefficients, a list of scene transitions may be generated in real-time. The list of scene transitions forms the basis for a content-based reverse user control within the playback application. The content-based reverse user control enables a more natural mechanism for reversing through video content, providing the user with a superior overall viewing experience.

Claims

1. A system for generating a list of content-based scene transitions within a stream of video data, the system comprising:

a first video decoder engine configured to receive the video data and to generate a first plurality of video frames based on the video data;

a first frame buffer for storing the first plurality of video frames;

a frame correlation engine configured to generate at least one correlation coefficient based on a comparison between at least two video frames in the first plurality of video frames;

a graphics processing unit-to-central processing unit (GPU-to-CPU) coefficient data transport configured to transmit the at least one correlation coefficient to a playback application for determining whether a scene boundary exists between any of the at least two video frames based on the at least one correlation coefficient and for generating an entry in a list of scene transitions when a scene boundary exists;

a second video decoder engine configured to receive the video data and to generate a second plurality of video frames based on the video data;

a second frame buffer for storing the second plurality of video frames; and

a display engine configured to generate a video output signal to display the second plurality of video frames,

wherein the second video decoder engine is associated with a reverse mode or a fast-forward mode, and wherein the reverse mode or the fast-forward mode is associated with the at least one correlation coefficient, and

wherein the first plurality of video frames and the second plurality of video frames represent two independently read versions of the video data.

2. The system of claim 1, wherein a first execution thread runs a first process that includes generating the at least one correlation coefficient using the first video decoder engine and the frame correlation engine, and a second execution thread runs a second process for displaying the video data using the second video decoder engine and the display engine.

3. The system of claim 2, wherein the frame correlation engine includes a pixel difference engine that applies a pixel difference function between each pixel in a first video frame of the at least two video frames and a corresponding pixel in a second video frame of the at least two video frames to generate a set of difference pixel values.

4. The system of claim 3, wherein the pixel difference function is applied using the luminance value associated with each pixel.

5. The system of claim 3, wherein the frame correlation engine further includes a pixel difference summing engine that adds together the difference pixel values to produce the at least one correlation coefficient.

6. The system of claim 2, wherein the frame correlation engine includes a frame histogram engine configured to generate a first histogram of the luminance values associated with the pixels in a first video frame of the at least two video frames and a second histogram of the luminance values associated with the pixels in a second video frame of the at least two video frames.

7. The system of claim 6, wherein the frame correlation engine further includes a histogram difference engine that applies a histogram difference function between an element in the first histogram and a corresponding element in the second histogram to generate a set of histogram difference values, and a histogram difference summing engine that adds together the histogram difference values to generate the at least one correlation coefficient.

8. The system of claim 2, wherein the frame correlation engine includes a pixel luminance correlation engine configured to:

apply a pixel difference function between the luminance value of each pixel in a first video frame of the at least two video frames and the luminance value of a corresponding pixel in a second video frame of the at least two video frames to generate a set of luminance difference values; and

add together the luminance difference values to produce a first correlation coefficient.

9. The system of claim 8, wherein the frame correlation engine further includes a luminance histogram correlation engine configured to:

generate a first histogram of the luminance values associated with the pixels in the first video frame and a second histogram of the luminance values associated with the pixels in the second video frame;

apply a histogram difference function between an element in the first histogram and a corresponding element in the second histogram to generate a set of histogram difference values; and

add together the histogram difference values to generate a second correlation coefficient.

10. The system of claim 2, further comprising a data transport element configured to transfer the at least one correlation coefficient to a central processing unit (CPU), wherein the first process also includes transporting the correlation coefficient to the CPU.

11. The system of claim 10, wherein a playback application executes on the CPU and is configured to determine whether a scene boundary exists between any of the at least two video frames based on the at least one correlation coefficient and to generate an entry in a list of scene transitions when a determination is made that a scene boundary exists, and wherein the first process also includes determining whether a scene boundary exists and generating the entry in the list of scene transitions.

12. The system of claim 11, wherein the entry indicates the time at which the scene boundary is located within the video data.

13. The system of claim 12, wherein the playback application is configured to receive user input to view the video data in the reverse mode or the fast-forward mode that allows the user to view the video data beginning at the scene boundary.

14. The system of claim 13, wherein the display engine is configured to generate a video output signal to display the video data beginning at the scene boundary based on a playback time control signal generated by the playback application in response to the user input.

15. A computing device configured to generate a list of content-based scene transitions within a stream of video data, the computing device comprising:

a graphics processing unit (GPU) configured to;

receive the video data and generate a first plurality of video frames based on the video data;

generate at least one correlation coefficient based on a comparison between at least two video frames in the first plurality of video frames, wherein the at least one correlation coefficient is used to determine whether a scene boundary exists between any of the at least two video frames;

transmit the at least one correlation coefficient to a playback application via a graphics processing unit-to-central processing unit (GPU-to-CPU) coefficient data transport;

generate a second plurality of video frames based on the video data, and

generate a video output signal to display the second plurality of video frames; and

a central processing unit (CPU) that, when executing the playback application, is configured to;

determine whether a scene boundary exists between any of the at least two video frames based on the at least one correlation coefficient; and

generate an entry in a list of scene transitions when a scene boundary exists,

wherein the second plurality of video frames is associated with a reverse mode or a fast-forward mode, and wherein the reverse mode or the fast-forward mode is associated with the at least one correlation coefficient, and

wherein the first plurality of video frames and the second plurality of video frames represent two independently read versions of the video data.

16. The computing device of claim 15, wherein a first execution thread runs a first process that includes generating the at least one correlation coefficient and generating the entry in the list of scene transitions based on the at least one correlation coefficient, and a second execution thread runs a second process for displaying the video data.

17. The computing device of claim 15, wherein a playback application executes on the CPU and is configured to receive user input to view the video data in the reverse mode or the fast-forward mode and to generate a playback time control signal in response to the user input that causes the GPU to display the video data beginning at the scene boundary.

18. The computing device of claim 15, wherein the GPU computes the at least one correlation coefficient using a pixel difference function or a histogram difference function.

19. A method for generating a list of content-based scene transitions within a stream of video data, the method comprising:

receiving the video data and generating a first plurality of video frames based on the video data;

generating at least one correlation coefficient based on a comparison between at least two video frames in the first plurality of video frames, wherein the at least one correlation coefficient is used to determine whether a scene boundary exists between any of the at least two video frames;

transmitting the at least one correlation coefficient to a playback application via a graphics processing unit-to-central processing unit (GPU-to-CPU) coefficient data transport;

generating a second plurality of video frames based on the video data;

generating a video output signal to display the second plurality of video frames;

causing a central processing unit (CPU) to determine whether a scene boundary exists between any of the at least two video frames based on the at least one correlation coefficient; and

causing the CPU to generate an entry in a list of scene transitions when a scene boundary exists;

wherein the second plurality of video frames is associated with a reverse mode or a fast-forward mode, and wherein the reverse mode or the fast-forward mode is associated with the at least one correlation coefficient, and

wherein the first plurality of video frames and the second plurality of video frames represent two independently read versions of the video data.

20. The method of claim 19, wherein the steps of generating the first plurality of video frames and generating the correlation coefficient are performed at least partially in parallel with the steps of generating the second plurality of video frames and generating the video output signal.