Method and apparatus for shot detection in video streaming

• US 8,654,848 B2
• Filed: 10/03/2006
• Issued: 02/18/2014
• Est. Priority Date: 10/17/2005
• Status: Expired due to Fees

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Tian, Tao, Walker, Gordon Kent, Raveendran, Vijayalakshmi R.
• Primary Examiner(s): Dastouri, Mehrdad
• Assistant Examiner(s): RAHAMAN, MOHAMMED S
• Application Number:
  US11/538,421
• Publication Number:
  US 20070160128A1
• Time in Patent Office: 2,695 Days
• Field of Search: 375/240.16, 348/700
• US Class Current: 375/240.16
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/114 Adapting the group of pictures [GOP] structure, e.g. number of B-frames between two anchor frames H04N19/107 takes precedence
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/142 Detection of scene cut or scene change
• CPC Class Codes: H04N 19/159 Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
• CPC Class Codes: H04N 19/177 the unit being a group of pictures [GOP]
• CPC Class Codes: H04N 19/179 the unit being a scene or a shot
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/87 involving scene cut or scene change detection in combination with video compression
• CPC Class Codes: H04N 5/147 Scene change detection

Abstract

The invention comprises devices and methods for processing multimedia data. Such methods can include obtaining at least one metric indicative of a difference for a selected frame and adjacent frames in a plurality of video frames, the metric comprising bi-directional motion information and luminance difference information, determining a shot event in the selected frame based on the at least one metric, and adaptively encoding the selected frame based on the shot event. An apparatus for processing a multimedia data can include a motion compensator configured to obtain metrics indicative of a difference between adjacent frames of a plurality of video frames, said metrics comprising bi-directional motion information and luminance information, a shot classifier configured to determine a shot event in the plurality of video frames based on said metrics, and an encoder configured to adaptively encode the plurality of frames based on the shot event.

Claims

1. A method of processing multimedia data, the method comprising:

obtaining at least one metric indicative of a difference between a selected frame and frames temporally adjacent to the selected frame in a plurality of video frames, said at least one metric comprising bi-directional motion information and luminance difference information associated with the selected frame and the frames temporally adjacent to the selected frame;

determining a shot event associated with the selected frame based on said at least one metric, wherein said determining a shot event comprises determining that the selected frame contains at least one camera flashlight if the selected frame is not an abrupt frame or a slow change frame, an average luminance of the selected frame minus an average luminance of a temporally adjacent frame of the selected frame is equal to or exceeds a first threshold value, and a forward difference metric and a backward difference metric associated with the selected frame are both equal to or exceed a second threshold value; and

adaptively encoding the selected frame based on the shot event.

2. The method of claim 1, wherein obtaining at least one metric comprises calculating the at least one metric.

3. The method of claim 1, wherein the shot event indicates that the selected frame is an abrupt scene change, and wherein adaptively encoding comprises encoding the selected frame as an I-frame.

4. The method of claim 1, wherein the shot event indicates the selected frame is a portion of a plurality of frames comprising a slow scene change, and wherein adaptively encoding comprises encoding the selected frame as either a P-frame or a B-frame.

5. The method of claim 1, wherein adaptively encoding comprises identifying the selected frame as requiring special processing.

6. The method of claim 5, wherein the special processing comprises skipping the selected frame such that the selected frame is not encoded.

7. The method of claim 5, wherein the special processing comprises replicating a frame temporally adjacent to the selected frame and substituting the replicated frame for the selected frame.

8. The method of claim 1, wherein the shot event indicates the selected frame comprises an abrupt scene change, a portion of a slow scene change, or at least one camera flashlight.

9. The method of claim 1, wherein adaptively encoding comprises encoding the selected frame as an I-frame if the shot event does not indicate the selected frame comprises abrupt scene change, a portion of a slow scene change, or at least one camera flashlight.

10. The method of claim 1, wherein adaptively encoding comprises dropping the selected frame if the shot event does not indicate the selected frame comprises an abrupt scene change, a portion of a slow scene change, or at least one camera flashlight.

11. The method of claim 1, wherein the at least one metric comprises a frame difference metric, and wherein determining a shot event comprises determining the selected frame is an abrupt scene if the frame difference metric is greater than a third threshold value T₃.

12. The method of claim 11, wherein the frame difference metric for the selected frame is based on a ratio of a first difference metric and a second difference metric, wherein the first difference metric comprises a ratio of differences between the selected frame and a previous temporally adjacent frame, and the selected frame and a next temporally adjacent frame, and the second difference metric comprises a ratio of differences between the previous frame and a frame temporally adjacent and previous to the previous frame, and the differences between the previous frame and the selected frame.

13. The method of claim 11, wherein the frame difference metric for the selected frame comprises a ratio of γ

_C and γ

_P, where γ

_C is a ratio of a forward and a backward difference metric for the current selected frame, and γ

_P, is a ratio of a forward and a backward difference metric for a temporally adjacent and previous frame to the current selected frame.

14. The method of claim 11, wherein the frame difference metric is based on a luminance histogram difference associated with a frame temporally previous to the selected frame and the selected frame.

15. The method of claim 11, wherein determining a shot event comprises determining the selected frame to be a portion of a slow scene change if the selected frame is not determined to be an abrupt scene change and said frame difference metric indicates a difference between the selected frame and adjacent frames is less than the third threshold value T₃ and is greater than or equal to a fourth threshold value T₄.

16. The method of claim 1, wherein said determining a shot event comprises evaluating luminance difference information of the selected frame against at least one threshold value.

17. The method of claim 1, wherein the shot events are classified into three or more shot types.

18. The method of claim 1, wherein obtaining at least one metric comprises performing bi-directional motion compensation to generate bi-directional motion information, and wherein said bi-directional motion information and luminance difference information are determined for the selected frame based on the bi-directional motion compensation.

19. The method of claim 1, wherein determining a shot event comprises evaluating the at least one metric against a plurality of threshold values.

20. An apparatus for processing a multimedia data, comprising:

a motion compensator configured to obtain at least one metric indicative of a difference between a selected frame and frames temporally adjacent to the selected frame in a plurality of video frames, said at least one metric comprising bi-directional motion information and luminance information;

a shot classifier configured to determine shot events associated with the selected frame based on said at least one metric, wherein the shot classifier is further configured to determine that the selected frame contains at least one camera flashlight if the selected frame is not an abrupt frame or a slow change frame, an average luminance of the selected frame minus an average luminance of a temporally adjacent frame of the selected frame is equal to or exceeds a first threshold value, and a forward difference metric and a backward difference metric associated with the selected frame are both equal to or exceed a second threshold value; and

an encoder configured to adaptively encode the selected frame based on the shot event.

21. An apparatus for processing multimedia data, comprising:

means for obtaining at least one metric indicative of a difference between a selected frame and frames temporally adjacent to the selected frame in a plurality of video frames, said at least one metric comprising bi-directional motion information and luminance difference information associated with the selected frame and the frames temporally adjacent to the selected frame;

means for determining a shot event associated with the selected frame based on said at least one metric, wherein said means for determining a shot event comprises means for classifying the selected frame as a camera flash scene if the selected frame is not an abrupt frame or a slow change frame, an average luminance of the selected frame minus an average luminance of a temporally adjacent frame of the selected frame is equal to or exceeds a first threshold value, and a forward difference metric and a backward difference metric associated with the selected frame are both equal to or exceed a second threshold value; and

means for adaptively encoding the selected frame based on the shot event.

22. The apparatus of claim 21, wherein the shot event indicates that the selected frame is an abrupt scene change, and wherein the adaptively encoding means comprise means for encoding the selected frame as an I-frame.

23. The apparatus of claim 21, wherein the shot event indicates the selected frame is a portion of a plurality of frames comprising a slow scene change, and wherein the adaptively encoding means comprise means for encoding the selected frame as either a P-frame or a B-frame.

24. The apparatus of claim 21, wherein the shot event indicates that the selected frame contains at least one camera flashlight, and wherein the adaptively encoding means comprise means for identifying the selected frame as requiring special processing.

25. The apparatus of claim 21, wherein the special processing comprises skipping or dropping the selected frame such that the selected frame is not encoded.

26. The apparatus of claim 25, wherein the special processing comprises means for replicating a frame temporally adjacent to the selected frame and substituting the replicated frame for the selected frame.

27. The apparatus of claim 21, wherein the shot event indicates the selected frame comprises an abrupt scene change, a portion of a slow scene change, or includes at least one camera flashlight.

28. The apparatus of claim 21, wherein the adaptively encoding means comprise means for encoding the selected frame as an I-frame if the shot event does not indicate the selected frame comprises an abrupt scene change, a portion of a slow scene change, or at least one camera flashlight.

29. The apparatus of claim 21, wherein adaptively encoding means comprise dropping the selected frame if the shot event does not indicate the selected frame comprises an abrupt scene change, a portion of a slow scene change, or at least one camera flashlights.

30. The apparatus of claim 21, wherein the at least one metric comprises a frame difference metric, and wherein the determining a shot event means comprise means for determining the selected frame is an abrupt scene if the frame difference metric is greater than a third threshold value T₃.

31. The apparatus of claim 30, wherein the frame difference metric for the selected frame is based on a ratio of a first difference metric and a second difference metric, wherein the first difference metric comprises a ratio of differences between the selected frame and a previous temporally adjacent frame, and the selected frame and a next temporally adjacent frame, and the second difference metric comprises a ratio of differences between the previous frame and a frame temporally adjacent and previous to the previous frame, and the differences between the previous frame and the selected frame.

32. The apparatus of claim 30, wherein the frame difference metric for the selected frame comprises a ratio of γ

_C and γ

_P, wherein γ

_C is a ratio of a forward and a backward difference metric for the current selected frame, and γ

_P, is a ratio of a forward and a backward difference metric for a temporally adjacent and previous frame to the selected frame.

33. The apparatus of claim 30, wherein the frame difference metric is based on a luminance histogram difference associated with an adjacent frame temporally previous to the selected frame and the selected frame.

34. The apparatus of claim 30, wherein the determining a shot event means comprise means for classifying the selected frame as a slow scene change if the selected frame is not classified as an abrupt scene change and the frame difference metric indicates a difference between the selected frame and adjacent frames is less than the third threshold value T₃ and greater than or equal to a fourth threshold value T₄.

35. A non-transitory machine readable medium comprising instructions for processing multimedia data, wherein the instructions upon execution cause a machine to:

obtain at least one metric indicative of a difference between a selected frame and frames temporally adjacent to the selected frame in a plurality of video frames, said at least one metric comprising bi-directional motion information and luminance difference information associated with the selected frame and the frames temporally adjacent to the selected frame;

determine a shot event associated with the selected frame based on said at least one metric, wherein the determination of a shot event comprises determining that the selected frame contains at least one camera flashlight if the selected frame is not an abrupt frame or a slow change frame, an average luminance of the selected frame minus an average luminance of a temporally adjacent frame of the selected frame is equal to or exceeds a first threshold value, and a forward difference metric and a backward difference metric associated with the selected frame are both equal to or exceed a second threshold value; and

adaptively encode the selected frame based on the shot event.

36. A processor for processing multimedia data, said processor comprising a configuration to

obtain at least one metric indicative of a difference between a selected frame and frames temporally adjacent to the selected frame in a plurality of video frames, said at least one metric comprising bi-directional motion information and luminance difference information associated with the selected frame and the frames temporally adjacent to the selected frame;

determine a shot event associated with the selected frame based on said at least one metric, wherein said processor comprises a further configuration to determine the shot event based on determining that the selected frame contains at least one camera flashlight if the selected frame is not an abrupt frame or a slow change frame, an average luminance of the selected frame minus an average luminance of a temporally adjacent frame of the selected frame is equal to or exceeds a first threshold value, and a forward difference metric and a backward difference metric associated with the selected frame are both equal to or exceed a second threshold value; and

adaptively encode the selected frame based on the shot event.