DETECTION OF ACTION FRAMES OF A VIDEO STREAM

• US 20150364158A1
• Filed: 06/02/2015
• Published: 12/17/2015
• Est. Priority Date: 06/16/2014
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Gupte, Ajit, Rao, Ajit Venkat, Baheti, Pawan Kumar, Jagannathan, Padmapriya, Srinivasamurthy, Naveen, Kumar, Sanjeev, Acharya, Hemanth
• Granted Patent: US 9,715,903 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/3081 used signal is a video-frame or a video-field (P.I.P)
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 9/8047 using transform coding
• CPC Class Codes: H04N 9/806 with processing of the sound signal

Abstract

A method includes receiving, at a device, a plurality of image frames corresponding to a video stream. The plurality of image frames include a first image frame having a first resolution and a second image frame having a second resolution that is lower than the first resolution. The method also includes detecting, at the device, a trigger by analyzing the second image frame. The method further includes designating, at the device, the first image frame as an action frame based on the trigger.

Claims

1. A method comprising:

receiving, at a device, a plurality of image frames corresponding to a video stream, the plurality of image frames including a first image frame having a first resolution and a second image frame having a second resolution that is lower than the first resolution;

detecting, at the device, a trigger by analyzing the second image frame; and

designating, at the device, the first image frame as an action frame based on the trigger.

2. The method of claim 1, wherein the first image frame represents a first camera view that is substantially the same as a second camera view represented by the second image frame, and wherein the first camera view corresponds to a viewing angle, a viewing distance, or both, of a camera.

3. The method of claim 1, wherein the first image frame has a first timestamp indicating a first time, wherein the second image frame has a second timestamp indicating a second time, and wherein a difference between the first time and the second time satisfies a threshold.

4. The method of claim 3, further comprising:

designating, at the device, one or more other image frames of the plurality of image frames as action frames based on the trigger, wherein differences between a time indicated by a timestamp of each of the one or more other image frames and the second time satisfies the threshold;

storing, at the device, the one or more other image frames in a first memory after receiving the one or more other image frames; and

copying, at the device, the one or more other image frames from the first memory to a second memory based on determining that the one or more other image frames are designated as action frames.

5. The method of claim 1 further comprising:

determining, at the device, first data based on the plurality of image frames, the first data indicative of motion of one or more first objects; and

determining, at the device, second data based on the plurality of image frames, the second data indicative of motion of one or more second objects,

wherein each of the first objects is represented in the second image frame by fewer than a first number of pixels,

wherein each of the second objects is represented in the second image frame by at least the first number of pixels, and

wherein the trigger is detected based on the first data satisfying a first threshold, the second data satisfying a second threshold, or both.

6. The method of claim 5, wherein the first data indicates a motion vector length of at least one set of macro-blocks of the second image frame, wherein each of the at least one set of macro-blocks corresponds to textured areas of the second image frame, and wherein the second data indicates at least one connected component size that is determined by performing connected component analysis on the second image frame.

7. The method of claim 1, further comprising:

generating, at the device, motion data by analyzing the second image frame, the motion data indicative of motion of at least one object;

receiving audio data at the device, the audio data corresponding to the video stream; and

updating, at the device, a trigger detection threshold based on a characteristic of the audio data,

wherein the trigger is detected based on the motion data satisfying the trigger detection threshold.

8. The method of claim 1, further comprising determining, at the device, cumulative statistics for each region of a plurality of regions of the second image frame, wherein the trigger is detected based on the cumulative statistics, and wherein the cumulative statistics include at least one of region intra macro-block (MB) fractions and variance of cumulative motion.

9. The method of claim 8, further comprising:

determining a region motion metric for each region of the plurality of regions based on motion vectors of MBs included in the region, wherein the second image frame includes a plurality of MBs;

determining global motion values of the second image; and

determining a plurality of cumulative motion vectors for the plurality of regions, each cumulative motion vector of the plurality of cumulative motion vectors determined based on a region motion metric of a corresponding region and the global motion values,

wherein a first variance of cumulative motion vectors for a first region of the plurality of regions is determined based on a first plurality of cumulative motion vectors for the first region.

10. The method of claim 8, further comprising determining a region intra MB fraction for each region of the plurality of regions based on intra MB fractions for one or more MBs included in the region, wherein the second image frame includes a plurality of MBs, and wherein a particular intra MB fraction for each MB of the one or more MBs is determined based on first characteristics of the MB and second characteristics of a reference MB of the plurality of MBs.

11. The method of claim 1, further comprising:

receiving, at the device, third image frames at the device;

determining, at the device, at least one of a head direction of a user or an eye direction of the user based on the third image frames; and

determining, at the device, a region of interest of the second image frame based on at least one of the head direction or the eye direction,

wherein the trigger is detected based on the region of interest,

wherein the plurality of image frames are received via a first camera, and

wherein the third image frames are received via a second camera that is distinct from the first camera.

12. The method of claim 11, wherein the trigger is detected based on cumulative mode statistics of the region of interest, motion data indicative of motion of an object that is represented in the region of interest, or a combination thereof.

13. The method of claim 1, wherein the first image frame is captured by a camera, and wherein the second image frame is generated from the first image frame.

14. The method of claim 1, further comprising:

storing, at the device, the first image frame in a first memory after receiving the first image frame; and

copying, at the device, the first image frame from the first memory to a second memory based on determining that the first image frame is designated as the action frame.

15. The method of claim 14, further comprising:

determining a frame score of the first image frame, wherein the second memory stores multiples image frames in a plurality of groups, and wherein a group of the plurality of groups includes the first image frame;

updating a group score of the group based on the frame score;

selecting a particular image frame of a particular group of the plurality of groups based on a particular frame score of the particular image frame, a particular group score of the particular group, or both; and

deleting the particular image frame from the second memory,

wherein the first memory includes a first-in first-out (FIFO) buffer, and wherein the second memory has a larger storage capacity than the first memory.

16. An apparatus comprising:

a memory configured to store a plurality of image frames corresponding to a video stream;

a first object detector configured to determine whether a first image frame of the plurality of image frames includes a first object;

a first object motion analyzer configured to, when the first object detector determines that the first image frame includes the first object, analyze the first image frame to generate first motion data indicative of motion of the first object;

a trigger detector configured to detect a trigger when the first motion data satisfies a first object motion threshold; and

a frame designator configured to designate a second image frame of the plurality of image frames as an action frame based on the trigger.

17. The apparatus of claim 16, wherein the first motion data indicates a connected component size that is determined by performing connected component analysis of the first image frame.

18. The apparatus of claim 16, further comprising:

a second object detector configured to determine whether the first image frame includes a second object; and

a second object motion analyzer configured to, when the second object detector determines that the first image frame includes the second object, analyze the first image frame to generate second motion data indicative of motion of the second object,

wherein the trigger detector is further configured to detect the trigger based on determining that the second motion data satisfies a second object motion threshold.

19. The apparatus of claim 18, wherein the second motion data indicates a motion vector length of a set of macro-blocks of the first image frame, and wherein the set of macro-blocks corresponds to textured areas of the first image frame.

20. The apparatus of claim 18, wherein the first object is represented in the first image frame by a first number of pixels, and wherein the second object is represented in the first image frame by a second number of pixels that is less than the first number of pixels.

21. The apparatus of claim 16, further comprising:

a microphone configured to receive audio data corresponding to the video stream; and

a threshold adjuster configured to update the first object motion threshold based on a characteristic of the audio data.

22. The apparatus of claim 21, wherein the first image frame includes a first timestamp indicating a first time, wherein the characteristic of the audio data is associated with a second timestamp indicating a second time, wherein the second time is subsequent to the first time, and wherein a difference between the second time and the first time satisfies a threshold.

23. The apparatus of claim 16, further comprising:

a region of interest identifier configured to;

determine at least one of a head direction of a user or an eye direction of the user based on third image frames; and

determine a region of interest of the first image frame based on at least one of the head direction or the eye direction,

wherein the trigger is detected based on the region of interest,

wherein the plurality of image frames are received via a first camera, and

wherein the third image frames are received via a second camera that is distinct from the first camera.

24. The apparatus of claim 23,

wherein the first object detector is configured to determine whether the region of interest includes the first object, and

wherein the first motion data is generated by analyzing the region of interest when the region of interest includes the first object.

25. The apparatus of claim 16, further comprising a processor configured to:

store the first image frame in a first buffer after receiving the first image frame;

store the second image frame in a second buffer after receiving the second image frame; and

copy the second image frame from the second buffer to a second memory based on determining that the second image frame is designated as the action frame.

26. An apparatus comprising:

means for storing a plurality of image frames corresponding to a video stream;

first means for analyzing a first image frame of the plurality of image frames to generate first motion data indicative of motion of a first object;

means for determining that the first image frame includes the first object based on determining that the first motion data satisfies a first object motion threshold;

means for detecting a trigger when the first image frame includes the first object; and

means for designating a second image frame of the plurality of image frames as an action frame based on the trigger.

27. The apparatus of claim 26, further comprising:

second means for analyzing the first image frame to generate second motion data indicative of motion of a second object; and

means for determining that the first image frame includes the second object when the second motion data satisfies a second object motion threshold,

wherein the means for detecting is further configured to detect the trigger when the first image frame includes the second object.

28. The apparatus of claim 26, wherein the means for storing, the first means for analyzing, the means for determining, the means for detecting, and the means for designating are integrated into at least one of a mobile device, a communications device, a computer, an entertainment unit, a navigation device, a personal digital assistant (PDA), a music player, a video player, a decoder, or a set top box.

29. A computer-readable storage device storing instructions that, when executed by a processor, cause the processor to perform operations comprising:

receiving a plurality of image frames corresponding to a video stream, the plurality of image frames including a first image frame having a first resolution and a second image frame having a second resolution that is lower than the first resolution, wherein the first image frame has a first timestamp indicating a first time, wherein the second image frame has a second timestamp indicating a second time, and wherein a difference between the first time and the second time satisfies a threshold;

detecting a trigger by analyzing the second image frame; and

designating the first image frame as an action frame based on the trigger.

30. The computer-readable storage device of claim 29, wherein the operations further comprise:

receiving audio data corresponding to the video stream; and

updating a trigger detection threshold based on a characteristic of the audio data, wherein the trigger is detected based on the trigger detection threshold.