METHOD AND SYSTEM FOR MOVING OBJECT DETECTION WITH SINGLE CAMERA

• US 20170140231A1
• Filed: 11/11/2016
• Published: 05/18/2017
• Est. Priority Date: 11/13/2015
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: CHEN, Sheng, AYVACI, Alper
• Granted Patent: US 10,019,637 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/2113 by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 2201/07 Target detection

Abstract

Disclosed are systems and methods for detecting moving objects. A computer-implemented method for detecting moving objects comprises obtaining a streaming video captured by a camera; extracting an input image sequence including a series of images from the streaming video; tracking point features and maintaining a set of point trajectories for at least one of the series of images; measuring a likelihood for each point trajectory to determine whether it belongs to a moving object using constraints from multi-view geometry; and determining a conditional random field (CRF) on an entire frame to obtain a moving object segmentation.

Claims

1. A computer-implemented method for detecting moving objects, comprising:

obtaining a streaming video captured by a camera;

extracting an input image sequence including a series of images from the streaming video;

tracking point features and maintaining a set of point trajectories for at least one of the series of images;

measuring a likelihood for each point trajectory to determine whether it belongs to a moving object using constraints from multi-view geometry; and

determining a conditional random field (CRF) on an entire frame to obtain a moving object segmentation.

2. The method of claim 1, wherein the camera comprises a monocular camera.

3. The method of claim 1, wherein the constraints from multi-view geometry comprise at least one of:

an epipolar constraint between two-view and trifocal constraints from three-view.

4. The method of claim 3, wherein the epipolar constraint is calculated based at least in part on an epipolar moving objectness score for a pair of point correspondence as follows:

φ

(x_i^τ

, x_i<sup>τ

′</sup>

)=d_pl(F_τ

<sup>τ

′</sup>

x_i<sup>τ

′</sup>

, x_i^τ

)+d_pl(x_i<sup>τ

′</sup>

, x_i^τ

F_τ

<sup>τ

′</sup>

),

where F_τ

<sup>τ

′</sup>

x_i<sup>τ

′</sup>

and x_i^τ

F_τ

<sup>τ

′</sup>

define relative epipolar lines in each view and a function d_pl(·

) computes a point to line distance.

5. The method of claim 4, wherein the pair of point correspondence is determined based on an optical flow between consecutive frames.

6. The method of claim 4, further comprising determining a weighted average of all epipolar moving objectness scores of a trajectory as follows:

7. The method of claim 4, further comprising determining another epipolar moving objectness score as follows:

8. The method of claim 3, wherein the trifocal constraints from three-view are determined based at least in part on a trifocal moving objectness score defined as follows:

γ

(x_i^m, x_iⁿ, x_i^p)=d_pp(x_i^p, {circumflex over (x)}_i^p)

where {circumflex over (x)}^t″

is the estimated location in the third view using x^t and x^t′

and d_pp(·

) computes the Euclidean distance.

9. The method of claim 8, further comprising determining a weighted average of all trifocal moving objectness scores to account for a history for each trajectory as follows:

10. The method of claim 1, further comprising, for each frame T:

computing an optical flow and a point trajectory;

estimating fundamental matrices and a trifocal tensor;

computing an epipolar moving objectness score and a trifocal moving objectness score for each trajectory; and

forming the CRF on superpiexels to determine moving labels.

11. A system for detecting moving objects, the system comprising:

means for obtaining a streaming video captured by a camera;

means for extracting an input image sequence including a series of images from the streaming video;

means for tracking point features and means for maintaining a set of point trajectories for at least one of the series of images;

means for measuring a likelihood for each point trajectory to determine whether it belongs to a moving object using constraints from multi-view geometry; and

means for determining a conditional random field (CRF) on an entire frame to obtain a dense moving object segmentation.

12. The system of claim 11, wherein the constraints from multi-view geometry comprise at least one of:

an epipolar constraint between two-view and trifocal constraints from three-view.

13. The system of claim 12, wherein the epipolar constraint is calculated based at least in part on an epipolar moving objectness score for a pair of point correspondence as follows:

)

φ

(x_i^τ

, x_i<sup>τ

′</sup>

)=d_pl(F_τ

<sup>τ

′</sup>

x_i<sup>τ

′</sup>

, x_i^τ

)+d_pl(x_i<sup>τ

′</sup>

, x_i^τ

F_τ

<sup>τ

′</sup>

),

where F_τ

<sup>τ

′</sup>

x_i<sup>τ

′</sup>

and x_i^τ

F_τ

<sup>τ

′</sup>

define relative epipolar lines in each view and a function d_pl(·

) computes a point to line distance.

14. The system of claim 13, further comprising means for determining a weighted average of all epipolar moving objectness scores of a trajectory as follows:

15. The system of claim 13, further comprising means for determining another epipolar moving objectness score as follows:

16. The system of claim 12, wherein the trifocal constraints from three-view is determined based at least in part on a trifocal moving objectness score as follows:

γ

(x_i^m, x_iⁿ, x_i^p)=d_pp(x_i^p, {circumflex over (x)}_i^p),

where {circumflex over (x)}^t″

is the estimated location in the third view using x^t and x^t′

and d_pp(·

) computes the Euclidean distance.

17. The system of claim 16, further comprising means for determining a weighted average of all trifocal moving objectness scores to account for a history for each trajectory as follows:

18. The system of claim 11, further comprising, for each frame T:

means for computing an optical flow and a point trajectory;

means for estimating fundamental matrices and a trifocal tensor;

means for computing an epipolar moving objectness score and a trifocal moving objectness score for each trajectory; and

means for forming the CRF on superpiexels to determine moving labels.

19. A non-transitory computer readable storage medium having control logic stored thereon for causing a computer to detect moving objects, the control logic comprising machine executable codes for:

obtaining a streaming video captured by a camera;

extracting an input image sequence including a series of images from the streaming video;

tracking point features and maintaining a set of point trajectories for at least one of the series of images;

measuring a likelihood for each point trajectory to determine whether it belongs to a moving object using constraints from multi-view geometry; and

determining a conditional random field (CRF) on an entire frame to obtain a dense moving object segmentation.

20. The non-transitory computer readable storage medium of claim 19, for each frame T, further comprising codes for:

computing an optical flow and a point trajectory;

estimating fundamental matrices and a trifocal tensor;

computing an epipolar moving objectness score and a trifocal moving objectness score for each trajectory; and

forming the CRF on superpiexels to determine moving labels.