Detection of Motion Activity Saliency in a Video Sequence

• US 20150222919A1
• Filed: 01/31/2014
• Published: 08/06/2015
• Est. Priority Date: 01/31/2014
• Status: Active Grant

• Current Assignee: HERE Global B.V. (Nokia Corporation)
• Original Assignee: HERE Global B.V. (Nokia Corporation)
• Inventors: Licata, Aaron
• Granted Patent: US 10,104,394 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G08B 13/19602 Image analysis to detect motion of the intruder, e.g. by frame subtraction
• CPC Class Codes: G08B 13/19604 involving reference image or background adaptation with time to compensate for changing conditions, e.g. reference image update on detection of light level change
• CPC Class Codes: G08B 13/1961 Movement detection not involving frame subtraction, e.g. motion detection on the basis of luminance changes in the image
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/513 Processing of motion vectors
• CPC Class Codes: H04N 19/521 for estimating the reliability of the determined motion vectors or motion vector field, e.g. for smoothing the motion vector field or for correcting motion vectors
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20

Abstract

A streaming video or a stored video is analyzed to identify salient features. The salient features are the more interesting portions of the video because salient features include the most motion. A motion vector field including a motion vector for multiple pixels in the video is generated. The motion vector field is partitioned into grid blocks, and each of the grid blocks is divided into a set of orientation ranges. The vectors of the motion vector field for each grid block are binned into the orientation ranges. A motion score is calculated based on a count of motion vectors for the orientation ranges. The section of the video having higher or the highest motion score is designated as a salient feature. Among other applications, salient features may be used for thumbnail images, highlight reels, or video editing.

Claims

1. A method comprising:

generating, using a processor, a motion vector field including a motion vector for each of a plurality of pixels in a video;

partitioning the motion vector field into a plurality of grid blocks;

binning motion vectors assigned to each of the plurality of grid blocks into a plurality of orientation ranges; and

calculating, using the processor, a motion score based on a count of motion vectors for at least one of the orientation ranges.

2. The method of claim 1, further comprising:

comparing the count of motion vectors in a first video frame to a corresponding count of motion vectors in a second frame.

3. The method of claim 1, further comprising:

comparing the count of motion vectors in a first set of video frames over a first time range to a corresponding count of motion vectors in a second set of video frames over a second time range.

4. The method of claim 1, wherein calculating the motion score comprises:

calculating a first difference between a first orientation range for a first frame and the first orientation range for a second frame;

calculating a second difference between a second orientation range for the first frame and the second orientation for the second frame; and

summing at least the first difference and the second difference as the motion score.

5. The method of claim 1, further comprising:

comparing the motion score to other motion scores for other frames of the video; and

identifying a highest motion score for the video.

6. The method of claim 5, further comprising:

generating metadata to designate a frame having the highest motion score.

7. The method of claim 5, further comprising:

displaying a frame having the highest motion score as a thumbnail image.

8. The method of claim 1, further comprising:

comparing the motion score and other motion scores for other frames of the video to a threshold motion score;

selecting one or more frames with motion scores that exceed the threshold motion score; and

discarding one or more frames with motion scores that are less than the threshold motion score.

9. The method of claim 1, further comprising:

receiving the video at a first resolution; and

downsampling the video to a second resolution prior to generating the motion vector field.

10. An apparatus comprising:

at least one processor; and

at least one memory including computer program code for one or more programs;

the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to at least perform;

calculating motion vector fields for time ranges in a video, wherein each of the motion vector fields includes motion vectors that describe movement of pixels in the video;

sorting the motion vectors into a plurality of orientation ranges;

performing a comparison of the motion vectors in pairs of subsequent time ranges; and

calculating a motion score for each pair of subsequent time ranges based on the comparison.

11. The apparatus of claim 10, wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to at least perform:

comparing the motion score for each of the subsequent time ranges to select a highest motion score; and

storing a flag with one or more frames of the video, wherein the flag is indicative of the highest motion score.

12. The apparatus of claim 11, wherein the flag is accessed to retrieve a thumbnail image of the video.

13. The apparatus of claim 10, wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to at least perform:

comparing the motion score for each of the subsequent time ranges to a threshold motion score; and

storing a flag with one or more frames of the video associated with motion scores that exceed the threshold motion score.

14. The apparatus of claim 13, wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to at least perform:

deleting at least a portion of the video associated with motion scores below the threshold motion score.

15. The apparatus of claim 13, wherein the flag defines advance points for advancing the video.

16. The apparatus of claim 10, wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to at least perform:

comparing the motion score for each of subsequent time ranges to identify a highlight time range with a highest average motion score; and

storing a flag with frames of the highlight time range, wherein the flag is indicative of the highest average motion score.

17. The apparatus of claim 10, wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to at least perform:

reducing a resolution of the video prior to calculating motion vector fields for the video.

18. A non-transitory computer readable medium including instructions that when executed are operable to:

collect a video comprising a series of video images;

perform a comparison of motion scores for the series of video images, wherein the motion scores are calculated from motion vectors for tracking movement in the series of video images that are organized by orientation; and

select one or more of the video images to represent the video based on the comparison of the motion scores.

19. The non-transitory computer readable medium of claim 18, the instructions configured to:

generate a motion vector field for a plurality of grid blocks forming the series of video images;

sort motion vectors assigned to each of the plurality of grid blocks into a plurality of orientation ranges; and

calculate the motion score based on a count of motion vectors for at least one of the orientation ranges.

20. The non-transitory computer readable medium of claim 18, the instructions configured to:

display one or more of the video images to represent the video based on the comparison of the motion scores.