Object position tracking using motion estimation

• US 9,659,380 B1
• Filed: 01/07/2016
• Issued: 05/23/2017
• Est. Priority Date: 01/07/2016
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Castellani, Marzia E., Ragusa, Roberto, Rea, Alessandro, Guarda, Roberto
• Primary Examiner(s): Rahmjoo, Manuchehr
• Application Number:
  US14/990,315
• Time in Patent Office: 502 Days
• Field of Search: None
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30236 Traffic on road, railway or crossing
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51

Abstract

A method includes tracking positions of object(s) in video frames, including: processing an initial frame of a set of frames of the video frames using feature extraction to identify locations of features of the object(s), obtaining a next frame of the set and applying a motion estimation algorithm as between the next frame and a prior frame to identify updated locations of the features in the next frame, where locations of the features as identified based on the prior frame are used as input to the motion estimation algorithm to identify the updated locations of the features in the next frame based one searching less than an entirety of the next frame. The tracking further includes recognizing occurrence of an event, halting the iteratively performing, and repeating, for at least one subsequent set of frames, the processing an initial frame and the using motion estimation.

Claims

1. A computer-implemented method comprising:

tracking position of at least one object in a plurality of video frames, the tracking comprising;

processing an initial frame of a set of frames of the plurality of video frames, the processing the initial frame using feature extraction to identify locations, in the initial frame, of features of the at least one object;

using motion estimation to track locations of the features in subsequent frames of the set of frames, comprising iteratively performing;

obtaining a next frame of the set of frames; and

applying a motion estimation algorithm as between the next frame and a prior frame of the set of frames to identify updated locations of the features in the next frame, wherein locations of the features as identified based on the prior frame are used as input to the motion estimation algorithm to identify the updated locations of the features in the next frame based on searching less than an entirety of the next frame;

recognizing occurrence of an event,

halting the iteratively performing based on recognizing the occurrence of the event, the halting defining the end of the set of frames; and

based on recognizing the occurrence of the event and the halting the iteratively performing, repeating, for each additional set of one or more subsequent sets of frames of the plurality of video frames, the processing an initial frame and the using motion estimation to track locations of the features, wherein the processing the initial frame of a respective additional set uses feature extraction to identify locations, in the initial frame of the additional set, of the features, and the using motion estimation to track locations of the features in subsequent frames of the additional set comprises iteratively performing the obtaining a next frame and the applying the motion detection algorithm as between the next frame and a prior frame of the additional set of frames.

2. The method of claim 1, wherein recognizing occurrence of the event comprises determining that a threshold number of subsequent frames of the set of frames have been processed to track the locations of the features in the subsequent frames.

3. The method of claim 1, wherein recognizing occurrence of the event comprises determining that a timeframe has lapsed since processing the initial frame of the set of frames.

4. The method of claim 1, wherein recognizing occurrence of the event comprises determining that an output of the motion detection algorithm is worse than a threshold value.

5. The method of claim 1, wherein the tracking position of the at least one object further comprises:

performing a scene change analysis at each iteration of the iteratively performing to detect whether a scene change in the plurality of video frames has occurred; and

recognizing occurrence of the event based on detecting that a scene change has occurred.

6. The method of claim 1, wherein the locations of the features define position of the at least one object, and wherein the repeating, for the one or more subsequent sets of frames, the processing the initial frame and the using the motion detection algorithm to track locations of the features provides tracking position of the at least one object across the plurality of video frames.

7. The method of claim 1, wherein at each iteration of the iteratively performing, the prior frame is the initial frame or is a frame immediately preceding the next frame.

8. The method of claim 1, wherein the features of the at least one object comprise a first set of features in a first frame area, wherein the method further identifies a second set of features, the second set of features being features of at least one other object, the at least one other object being in a second frame area, different from the first frame area, and wherein the method uses, for each subsequent frame of one or more subsequent frames of the set of frames, motion estimation in tracking the first set of features in the first frame area to track the at least one object and feature extraction in tracking the second set of features in the second frame area to track the at least one other object.

9. The method of claim 1, further comprising establishing, as part of an augmented reality application, an active element on a display of a user device, the active element overlaying an object, of the at least one object, in the plurality of video frames based on the tracking the position of the object in the plurality of video frames.

10. A computer system comprising:

a memory; and

a processor in communications with the memory, wherein the computer system is configured to perform a method, the method comprising;

tracking position of at least one object in a plurality of video frames, the tracking comprising;

processing an initial frame of a set of frames of the plurality of video frames, the processing the initial frame using feature extraction to identify locations, in the initial frame, of features of the at least one object;

using motion estimation to track locations of the features in subsequent frames of the set of frames, comprising iteratively performing;

obtaining a next frame of the set of frames; and

applying a motion estimation algorithm as between the next frame and a prior frame of the set of frames to identify updated locations of the features in the next frame, wherein locations of the features as identified based on the prior frame are used as input to the motion estimation algorithm to identify the updated locations of the features in the next frame based on searching less than an entirety of the next frame

recognizing occurrence of an event;

halting the iteratively performing based on recognizing the occurrence of the event, the halting defining the end of the set of frames; and

based on recognizing the occurrence of the event and the halting the iteratively performing, repeating, for each additional set of one or more subsequent sets of frames of the plurality of video frames, the processing an initial frame and the using motion estimation to track locations of the features, wherein the processing the initial frame of a respective additional set uses feature extraction to identify locations, in the initial frame of the additional set, of the features, and the using motion estimation to track locations of the features in subsequent frames of the additional set comprises iteratively performing the obtaining a next frame and the applying the motion detection algorithm as between the next frame and a prior frame of the additional set of frames.

11. The computer system of claim 10, wherein the features of the at least one object comprise a first set of features in a first frame area, wherein the method further identifies a second set of features, the second set of features being features of at least one other object, the at least one other object being in a second frame area, different from the first frame area, and wherein the method uses, for each subsequent frame of one or more subsequent frames of the set of frames, motion estimation in tracking the first set of features in the first frame area to track the at least one object and feature extraction in tracking the second set of features in the second frame area to track the at least one other object.

12. The computer system of claim 10, wherein the method further comprises establishing, as part of an augmented reality application, an active element on a display of a user device, the active element overlaying an object, of the at least one object, in the plurality of video frames based on the tracking the position of the object in the plurality of video frames.

13. A computer program product comprising:

a computer readable storage medium readable by a processor and storing instructions for execution by the processor for performing a method comprising;

tracking position of at least one object in a plurality of video frames, the tracking comprising;

processing an initial frame of a set of frames of the plurality of video frames, the processing the initial frame using feature extraction to identify locations, in the initial frame, of features of the at least one object;

using motion estimation to track locations of the features in subsequent frames of the set of frames, comprising iteratively performing;

obtaining a next frame of the set of frames; and

applying a motion estimation algorithm as between the next frame and a prior frame of the set of frames to identify updated locations of the features in the next frame, wherein locations of the features as identified based on the prior frame are used as input to the motion estimation algorithm to identify the updated locations of the features in the next frame based on searching less than an entirety of the next frame;

recognizing occurrence of an event;

halting the iteratively performing based on recognizing the occurrence of the event, the halting defining the end of the set of frames; and

based on recognizing the occurrence of the event and the halting the iteratively performing, repeating, for each additional set of one or more subsequent sets of frames of the plurality of video frames, the processing an initial frame and the using motion estimation to track locations of the features, wherein the processing the initial frame of a respective additional set uses feature extraction to identify locations, in the initial frame of the additional set, of the features, and the using motion estimation to track locations of the features in subsequent frames of the additional set comprises iteratively performing the obtaining a next frame and the applying the motion detection algorithm as between the next frame and a prior frame of the additional set of frames.

14. The computer program product of claim 13, wherein the features of the at least one object comprise a first set of features in a first frame area, wherein the method further identifies a second set of features, the second set of features being features of at least one other object, the at least one other object being in a second frame area, different from the first frame area, and wherein the method uses, for each subsequent frame of one or more subsequent frames of the set of frames, motion estimation in tracking the first set of features in the first frame area to track the at least one object and feature extraction in tracking the second set of features in the second frame area to track the at least one other object.