Detection of motion activity saliency in a video sequence

• US 10,104,394 B2
• Filed: 01/31/2014
• Issued: 10/16/2018
• Est. Priority Date: 01/31/2014
• Status: Active Grant

• Current Assignee: HERE Global B.V. (Nokia Corporation)
• Original Assignee: HERE Global B.V. (Nokia Corporation)
• Inventors: Licata, Aaron
• Primary Examiner(s): Hance, Robert J
• Application Number:
  US14/169,703
• Publication Number:
  US 20150222919A1
• Time in Patent Office: 1,719 Days
• CPC Class Codes: G08B 13/19602 Image analysis to detect motion of the intruder, e.g. by frame subtraction
• CPC Class Codes: G08B 13/19604 involving reference image or background adaptation with time to compensate for changing conditions, e.g. reference image update on detection of light level change
• CPC Class Codes: G08B 13/1961 Movement detection not involving frame subtraction, e.g. motion detection on the basis of luminance changes in the image
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/513 Processing of motion vectors
• CPC Class Codes: H04N 19/521 for estimating the reliability of the determined motion vectors or motion vector field, e.g. for smoothing the motion vector field or for correcting motion vectors
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20

Abstract

A streaming video or a stored video is analyzed to identify salient features. The salient features are the more interesting portions of the video because salient features include the most motion. A motion vector field including a motion vector for multiple pixels in the video is generated. The motion vector field is partitioned into grid blocks, and each of the grid blocks is divided into a set of orientation ranges. The vectors of the motion vector field for each grid block are binned into the orientation ranges. A motion score is calculated based on a count of motion vectors for the orientation ranges. The section of the video having higher or the highest motion score is designated as a salient feature. Among other applications, salient features may be used for thumbnail images, highlight reels, or video editing.

Claims

1. A method comprising:

generating, using a processor, a motion vector field including a motion vector for each of a plurality of pixels in a video;

partitioning the motion vector field into a plurality of grid blocks;

binning motion vectors assigned to each of the plurality of grid blocks into a plurality of orientation ranges;

calculating a grid block weight for each of the plurality of grid blocks based on a distance to a center of the video;

calculating, using the processor, a plurality of motion scores, wherein each of the plurality of motion scores is for a different frame of the video and is based on the grid block weight based on the distance to the center of the video and a count of motion vectors for at least one of the orientation ranges;

identifying a frame having the highest motion score for the video, the identifying based on comparing the motion scores for different frames of the video; and

generating data indicative of a thumbnail image for the frame of the video having the highest motion score.

2. The method of claim 1, wherein calculating the plurality of motion scores comprises:

comparing the count of motion vectors in a first set of video frames over a first time range to a corresponding count of motion vectors in a second set of video frames over a second time range.

3. The method of claim 1, wherein calculating the motion score comprises:

calculating a first difference between a first orientation range for a first frame and the first orientation range for a second frame;

calculating a second difference between a second orientation range for the first frame and the second orientation for the second frame; and

summing at least the first difference and the second difference as the motion score.

4. The method of claim 1, further comprising:

generating metadata to designate a frame having the highest motion score.

5. The method of claim 1, further comprising:

displaying the thumbnail image for the video.

6. The method of claim 1, further comprising:

comparing the motion scores for the frames of the video to a threshold motion score;

selecting one or more frames with motion scores that exceed the threshold motion score; and

discarding one or more frames with motion scores that are less than the threshold motion score.

7. The method of claim 1, further comprising:

receiving the video at a first resolution; and

downsampling the video to a second resolution prior to generating the motion vector field.

8. An apparatus comprising:

at least one processor; and

at least one memory including computer program code for one or more programs;

the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to at least perform;

calculating motion vector fields for time ranges of frames in a video, wherein each of the motion vector fields includes motion vectors that describe movement of pixels in the video;

partitioning the motion vectors into a plurality of grid blocks;

sorting the motion vectors within each of the plurality of grid blocks into a plurality of orientation ranges;

calculating a grid block weight for each of the plurality of grid blocks based on a distance to a center of the video;

applying the grid block weight to motion vectors as weighted motion vectors;

comparing the weighted motion vectors in pairs of subsequent time ranges to calculate a motion score for each pair of subsequent time ranges;

comparing the motion scores for each of the subsequent time ranges to select a frame having the highest motion score for the video; and

generating data indicative of a thumbnail image to represent the video, the data indicative of a thumbnail image generated using the frame having the highest motion score for the video.

9. The apparatus of claim 8, wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to at least perform:

storing a flag with one or more frames of the video, wherein the flag is indicative of the highest motion score.

10. The apparatus of claim 9, wherein the flag is accessed to generate the thumbnail image to represent the video.

11. The apparatus of claim 8, wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to at least perform:

comparing the motion score for each of the subsequent time ranges to a threshold motion score; and

storing a flag with one or more frames of the video associated with motion scores that exceed the threshold motion score.

12. The apparatus of claim 11, wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to at least perform:

deleting at least a portion of the video associated with motion scores below the threshold motion score.

13. The apparatus of claim 11, wherein the flag defines advance points for advancing the video.

14. The apparatus of claim 8, wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to at least perform:

comparing the motion score for each of subsequent time ranges to identify a highlight time range with a highest average motion score; and

storing a flag with frames of the highlight time range, wherein the flag is indicative of the highest average motion score.

15. The apparatus of claim 8, wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to at least perform:

reducing a resolution of the video prior to calculating motion vector fields for the video.

16. A non-transitory computer readable medium including instructions that when executed are operable to:

collect a video comprising a series of video frames;

generate motion vectors for tracking movement in the series of video frames, wherein each of the motion vectors is associated with one of a plurality of grid blocks forming the series of video frames;

calculate a grid block weight for each of the plurality of grid blocks based on a distance to a center of the video frames;

apply the grid block weight to motion vectors as weighted motion vectors;

sort weighted motion vectors assigned to each of the plurality of grid blocks into a plurality of orientation ranges;

calculate a motion score based on a count of weighted motion vectors for at least one of the orientation ranges;

perform a comparison of motion scores for each video frame of the series of video frames, wherein the motion scores are calculated from the weighted motion vectors; and

select one or more of the video frames to represent the video as a thumbnail image or a highlight reel, the selection based on the comparison of the motion scores, the selected one or more of the video frames having the highest motion score for the video.

17. The non-transitory computer readable medium of claim 16, the instructions configured to:

display the selected one or more of the video frames to represent the video as the thumbnail image or the highlight reel.