SYSTEM AND METHOD FOR VISUAL MOTION BASED OBJECT SEGMENTATION AND TRACKING

• US 20140334668A1
• Filed: 05/10/2013
• Published: 11/13/2014
• Est. Priority Date: 05/10/2013
• Status: Active Grant

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Palo Alto Research Center, Inc. (Xerox Holdings Corp.)
• Inventors: Saund, Eric
• Granted Patent: US 9,025,825 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/30236 Traffic on road, railway or crossing
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments

Abstract

The PMP Growth algorithm described herein provides for image tracking, segmentation and processing in environments where the camera system moves around a great deal, i.e., causing image jumps from one image frame to the next. It also is operative in systems where the objects themselves are making quick movements that alter their path. Attributes of the PMP Growth algorithm allow tracking systems using the PMP Growth algorithm to follow objects a long distance in a scene. This detection and tracking method is designed to track objects within a sequence of video image frames, and includes detecting keypoints in a current image frame of the video image frames, assigning local appearance features to the detected keypoints, establishing Point-Motion-Pairs between two successive image frames of the video image frames, and accumulating additional matches between image locations to form complete coherent motion object models of the objects being tracked. The segmentation aspect permits for the discovery of different coherently moving regions in the images.

Claims

1. A detection and tracking method for detecting and tracking objects within a sequence of video image frames, the method comprising;

detecting keypoints in a current image frame of the video image frames;

assigning local appearance features to the detected keypoints;

establishing Point-Motion-Pairs between the current image frame and a previous image frame of the video image frames, using the local appearance features;

using the Point-Motion-Pairs to form seed coherent motion object models; and

accumulating additional matches between image locations in the current image frame and the previous image frame to form complete coherent motion object models of the objects being tracked, wherein the steps are performed by at least one electronics processor.

2. The method according to claim 1 wherein the accumulating step includes establishing potential candidates merges for the seed coherent motion object models.

3. The method according to claim 2 further including using motion history to establish compatibility of motion object models for merging.

4. The method according to claim 1 wherein the step of establishing Point-Motion-Pairs includes pairing model points of the coherent motion objects from the previous image frame with image keypoints of the current image frame.

5. The method according to claim 4 further including using groups of the Point-Motion-Pairs to form local estimates of similarity transform parameters.

6. The method according to claim 5 further including constraining Point-Motion-Pairs to have consistent local patch appearance around keypoint locations.

7. The method according to claim 6 further including using a Symmetric Binary Robust Independent Elementary Features (SBRIEF) vector to represent and compare local image patch appearance.

8. A method for visual motion based object segmentation and tracking, the method comprising:

inputting a current image frame;

inputting state data carried over from a previous image frame;

detecting keypoint locations in the current image frame;

computing an appearance feature for each keypoint that can be compared with appearance features of keypoints and model points from the previous image frame;

performing Delaunay Triangulation on the current image frame keypoints, and on the model points of each of the previous image frame Similarity Transform Objects Models (STOMs);

generating a catalog of appearance correspondence matches between model points of the previous image frame and keypoint features of the current image frame;

forming, by use of the appearance correspondence matches, Point-Motion-Pairs between the previous image frame STOMs'"'"' model keypoints and the current image frame keypoints;

forming clusters of the Point-Motion-Pairs using compatible similarity transforms;

choosing a cluster of the Point-Motion-Pairs to seed the current image frame with STOMs;

growing the STOMs to include as many current image frame keypoints as possible;

merging consistent STOMs in the current image frame together;

finding consistent Point-Motion-Pair mappings between remaining image keypoints from the current image frame and keypoints that were left unexplained in the previous image frame to seed new STOMs; and

outputting a set of STOMs representing coherent motion regions between the current image frame and the previous image frame, and the set of remaining unexplained keypoints of the current image frame, wherein the steps are performed using at least one electronic processor.

9. The method according to claim 8 wherein following the step of merging, the step of finding consistent Point-Motion-Pair mappings between remaining image keypoints from the current image frame and keypoints that were left unexplained in the previous image frame to seed new STOMs includes:

performing Delaunay Triangulation to form a neighborhood graph among the previous image frame unexplained keypoints;

forming Point-Motion-Pairs between neighboring pairs of the previous image frame unexplained keypoints and the current image frame unexplained keypoints;

clustering the Point-Motion-Pairs by compatible similarity transform;

forming candidate STOM seeds by accumulating sets of compatible Point-Motion-Pair clusters;

growing the STOMs to accumulate additional unexplained current image frame keypoints;

recording merge candidates; and

executing the merge procedure among motion-compatible STOMs.

10. The method of claim 8 wherein the carried over state data includes a set of Similarity Transform Object Models (STOMs) and unexplained keypoints detected in the previous image frame and the STOMs each include a list of model keypoints comprising the set of image locations in the previous frame image that were ascribed to each STOM.

11. The method according to claim 8 wherein in the step of computing the appearance feature for each keypoint, an appearance of a local patch around each keypoint is represented as a binary feature vector describing the pattern of pixel lightnesses and/or colors in the neighborhood of the keypoint.

12. The method according to claim 8 wherein pairs of points of the Point-Motion-Pairs are assigned in a 1:

1 correspondence, consisting of a pair of points in the previous image frame in correspondence with a pair of points in the current image frame.

13. The method according to claim 12 wherein the correspondence defines the four parameters of a similarity transform, wherein the Point-Motion-Pair establishes a four degree of freedom transform of any object owning the points from the previous image frame to the current image frame.

14. The method of claim 8 wherein the matching of keypoints to locations using the appearance of the location is accomplished by use of a binary feature vector describing the pattern of pixel lightness and/or colors in the neighborhood of the keypoint.

15. The method of claim 8 wherein the binary feature vector is a Symmetric Binary Robust Independent Elementary Features (SBRIEF) feature descriptor.

16. The method according to claim 15 wherein the SBRIEF feature descriptor uses a particular layout pattern of sample locations with respect to a central location.

17. A system for detection and tracking objects within a sequence of video image frames, the system comprising:

at least one an electronic processor device configured to;

detect keypoints in a current image frame of the video image frames;

assign local appearance features to the detected keypoints;

establish Point-Motion-Pairs between the current image frame and a previous image frame of the video image frames, using the local appearance features;

use the Point-Motion-Pairs to form seed coherent motion object models; and

accumulate additional matches between image locations in the current image frame and the previous image frame to form complete coherent motion object models of the objects being tracked.

18. The system according to claim 17 wherein the accumulated matches are obtained by establishing potential candidates merges for the seed coherent motion object models.

19. The system according to claim 18 further including using motion history to establish compatibility of motion object models for merger.

20. The system according to claim 17 wherein the Point-Motion-Pairs are formed by a pair of model points in the previous image frame and a pair of model points in the current image frame.