Detection of action frames of a video stream

• US 9,715,903 B2
• Filed: 06/02/2015
• Issued: 07/25/2017
• Est. Priority Date: 06/16/2014
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Srinivasamurthy, Naveen, Acharya, Hemanth, Baheti, Pawan Kumar, Jagannathan, Padmapriya, Kumar, Sanjeev, Rao, Ajit Venkat, Gupte, Ajit
• Primary Examiner(s): Adams, Eileen
• Application Number:
  US14/728,047
• Publication Number:
  US 20150364158A1
• Time in Patent Office: 784 Days
• Field of Search: None
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/3081 used signal is a video-frame or a video-field (P.I.P)
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 9/8047 using transform coding
• CPC Class Codes: H04N 9/806 with processing of the sound signal

Abstract

A method includes receiving, at a device, a plurality of image frames corresponding to a video stream. The plurality of image frames include a first image frame having a first resolution and a second image frame having a second resolution that is lower than the first resolution. The method also includes detecting, at the device, a trigger by analyzing the second image frame. The method further includes designating, at the device, the first image frame as an action frame based on the trigger.

Claims

1. An apparatus comprising:

a memory configured to store a plurality of image frames corresponding to a video stream;

a first object detector configured to determine whether a first image frame of the plurality of image frames includes a first object;

a first object motion analyzer configured to, subsequent to the first object detector determining that the first image frame includes the first object, analyze the first image frame to generate first motion data corresponding to the first object;

a trigger detector configured to detect a trigger based on determining that the first motion data satisfies a first object motion threshold, the first object motion threshold determined based on data associated with one or more image frames, the one or more image frames prior to the first image frame in the plurality of image frames; and

a frame designator configured to designate a second image frame of the plurality of image frames as an action frame based on the trigger.

2. The apparatus of claim 1, wherein the first motion data indicates a connected component size that is determined by performing connected component analysis of the first image frame.

3. The apparatus of claim 1, further comprising:

a second object detector configured to determine whether the first image frame includes a second object; and

a second object motion analyzer configured to, subsequent to the second object detector determining that the first image frame includes the second object, analyze the first image frame to generate second motion data indicative of motion of the second object,

wherein the trigger detector is further configured to detect the trigger based on determining that the second motion data satisfies a second object motion threshold.

4. The apparatus of claim 3, wherein the second motion data indicates a motion vector length of a set of macro-blocks of the first image frame, and wherein the set of macro-blocks corresponds to textured areas of the first image frame.

5. The apparatus of claim 3, wherein the first object is represented in the first image frame by a first number of pixels, and wherein the second object is represented in the first image frame by a second number of pixels that is less than the first number of pixels.

6. The apparatus of claim 1, further comprising:

a microphone configured to receive audio data corresponding to the video stream; and

a threshold adjuster configured to update the first object motion threshold based on a characteristic of the audio data.

7. The apparatus of claim 6, wherein the first image frame includes a first timestamp indicating a first time, wherein the characteristic of the audio data is associated with a second timestamp indicating a second time, wherein the second time is subsequent to the first time, and wherein a difference between the second time and the first time satisfies a threshold.

8. The apparatus of claim 1, further comprising:

a region of interest identifier configured to;

determine at least one of a head direction of a user or an eye direction of the user based on third image frames; and

determine a region of interest of the first image frame based on at least one of the head direction or the eye direction,

wherein the first motion data is generated by analyzing the region of interest,

wherein the plurality of image frames are received via a first camera,

wherein the third image frames are received via a second camera that is distinct from the first camera.

9. The apparatus of claim 8,

wherein the first object detector is configured to determine whether the region of interest includes the first object, and

wherein the first motion data is generated by analyzing the region of interest subsequent to the first object detector determining that the region of interest includes the first object.

10. The apparatus of claim 1, further comprising a processor configured to:

store the first image frame in a first buffer after receiving the first image frame;

store the second image frame in a second buffer after receiving the second image frame; and

copy the second image frame from the second buffer to a second memory based on determining that the second image frame is designated as the action frame.

11. The apparatus of claim 1, wherein the memory, the first object detector, the first object motion analyzer, the trigger detector, and the frame designator are include in a mobile communication device.

12. The apparatus of claim 1, wherein the first image frame has a first timestamp indicating a first time, wherein the second image frame has a second timestamp indicating a second time, wherein a difference between the first time and the second time satisfies a threshold, wherein the frame designator is further configured to designate one or more other image frames of the plurality of image frames as action frames based on the trigger, and wherein differences between a time indicated by a timestamp of each of the one or more other image frames and the first time satisfy the threshold.

13. The apparatus of claim 1, wherein the first image frame represents a first camera view that is substantially the same as a second camera view represented by the second image frame, and further comprising:

a camera configured to provide the first camera view to capture the plurality of image frames corresponding to the video stream; and

a display device configured to output the first image frame, the display device coupled to the memory and to a display controller, wherein the first camera view corresponds to a viewing angle, a viewing distance, or both, of the camera, and wherein the camera is coupled to the display device.

14. The apparatus of claim 1, wherein the trigger is detected based on cumulative mode statistics of a region of interest, motion data indicative of motion of an object that is represented in the region of interest, or a combination thereof.

15. The apparatus of claim 1, further comprising:

a camera configured to capture the second image frame; and

a display device configured to output the second image frame, the display device coupled to the memory and to a display controller, wherein the first image frame is generated from the first image frame.

16. The apparatus of claim 1, further comprising a processor that includes the first object detector, the first object motion analyzer, the trigger detector, and the frame designator are include in a mobile communication device.

17. The apparatus of claim 16, wherein the processor is configured to copy one or more other image frames from the memory to a second memory based on determining that the one or more other image frames are designated as action frames.

18. The apparatus of claim 16, wherein the processor is configured to copy the second image frame from the memory to a second memory based on determining that the second image frame is designated as the action frame, and wherein the first image frame is a lower resolution frame than the second image frame.

19. The apparatus of claim 16, wherein the processor is configured to determine cumulative statistics for each region of a plurality of regions of the first image frame, wherein the trigger is detected based on the cumulative statistics, and wherein the cumulative statistics include at least one of region intra macro-block (MB) fractions and variance of cumulative motion.

20. The apparatus of claim 19, wherein the processor is further configured to determine a region motion metric for each region of the plurality of regions based on motion vectors of MBs included in the region, wherein the first image frame includes a plurality of MBs, wherein the processor is further configured to determine global motion values of the first image and to determine a plurality of cumulative motion vectors for the plurality of regions, each cumulative motion vector of the plurality of cumulative motion vectors determined based on a region motion metric of a corresponding region and the global motion values, and wherein a first variance of cumulative motion vectors for a first region of the plurality of regions is determined based on a first plurality of cumulative motion vectors for the first region.

21. The apparatus of claim 19, wherein the processor is further configured to determine a region intra MB fraction for each region of the plurality of regions based on intra MB fractions for one or more MBs included in the region, wherein the first image frame includes a plurality of MBs, and wherein a particular intra MB fraction for each MB of the one or more MBs is determined based on first characteristics of the MB and second characteristics of a reference MB of the plurality of MBs.

22. The apparatus of claim 16, wherein the processor is configured to determine a frame score of the second image frame, wherein a second memory stores multiples image frames in a plurality of groups, wherein a group of the plurality of groups includes the first image frame, and wherein the processor is further configured to update a group score of the group based on the frame score and to select a particular image frame of a particular group of the plurality of groups based on a particular frame score of the particular image frame, a particular group score of the particular group, or both.

23. The apparatus of claim 22, wherein the processor is configured to delete the particular image frame from the second memory, wherein the memory includes a first-in first-out (FIFO) buffer, and wherein the second memory has a larger storage capacity than the memory.

24. An apparatus comprising:

means for storing a plurality of image frames corresponding to a video stream;

first means for analyzing a first image frame of the plurality of image frames to generate first motion data corresponding to a first object;

means for determining that the first image frame includes the first object based on determining that the first motion data satisfies a first object motion threshold, the first object motion threshold determined based on data associated with one or more image frames, the one or more image frames prior to the first image frame in the plurality of image frames;

means for detecting a trigger based on determining that the first image frame includes the first object; and

means for designating a second image frame of the plurality of image frames as an action frame based on the trigger.

25. The apparatus of claim 24, further comprising:

second means for analyzing the first image frame to generate second motion data corresponding to a second object; and

means for determining that the first image frame includes the second object based on determining that the second motion data satisfies a second object motion threshold,

wherein the means for detecting is further configured to detect the trigger based on determining that the first image frame includes the second object.

26. The apparatus of claim 24, wherein the means for storing, the first means for analyzing, the means for determining, the means for detecting, and the means for designating are integrated into at least one of a mobile device, a communications device, a computer, an entertainment unit, a navigation device, a personal digital assistant (PDA), a music player, a video player, a decoder, or a set top box.

27. An apparatus comprising:

means for storing a plurality of image frames corresponding to a video stream;

means for determining whether a first image frame of the plurality of image frames includes a first object;

means for analyzing the first image frame to generate first motion data indicative of motion of the first object when the first image frame includes the first object;

means for detecting a trigger when the first motion data satisfies a first object motion threshold, the first object motion threshold determined based on data associated with one or more image frames, the one or more image frames prior to the first image frame in the plurality of image frames; and

means for designating a second image frame of the plurality of image frames as an action frame based on the trigger.

28. The apparatus of claim 27, wherein the first image frame has a first timestamp indicating a first time, wherein the second image frame has a second timestamp indicating a second time, and wherein a difference between the first time and the second time satisfies a threshold, and further comprising:

means for capturing the plurality of image frames;

means for receiving audio data corresponding to the video stream; and

means for updating a trigger detection threshold based on a characteristic of the audio data, wherein the trigger is detected based on the trigger detection threshold.