Method and apparatus for detecting independent motion in three-dimensional scenes
First Claim
1. A method for detecting independently moving objects in a video sequence of successive images of a three-dimensional (3D) scene at least two of the images being taken from respectively different camera positions, the method comprising the steps of:
- selecting a group of images including the at least two images taken from different camera positions;
calculating a set of two-dimensional (2D) view geometry constraints for the group of images;
testing the 2D view geometry constraints to determine if the imaged scene exhibits significant 3D characteristics;
if the imaged scene exhibits significant 3D characteristics, calculating a set of 3D shape constraints for the group of images, wherein the 3D shape constraints are constrained by the 2D view geometry constraints; and
if the imaged scene exhibits significant 3D characteristics, identifying areas in the group of images that are inconsistent with 3D constraints as areas corresponding to the moving objects.
2 Assignments
0 Petitions
Accused Products
Abstract
A system and method that detects independently moving objects in 3D scenes which are viewed under camera motion progressively applies constraints to the images to ensure the stability of the constraints. The system first calculates 2D view geometry constraints for a set of images. These constraints are tested to determine if the imaged scene exhibits significant 3D characteristics. If it does, then 3D shape constraints, are applied to the set of images. The 3D shape constraints are themselves constrained by the 2D view geometry constraints. The set of images is then tested to identify areas that are inconsistent with the 2D or 3D constraints. These areas correspond to the moving objects. The 2D view geometry constraints are calculated by computing a dominant image alignment for successive pairs of images and then computing constrained epipolar transformations for the two image pairs. This 2D view geometry is further refined based on a plurality of target point correspondences among the plurality of frames. The epipolar geometry for the point correspondence having a minimum median error is selected as the 2D view geometry of the scene. The 3D shape constraint is a parallax geometry that is calculated by iteratively minimizing errors in a parametric alignment of the images using an estimated parallax geometry.
-
Citations
21 Claims
-
1. A method for detecting independently moving objects in a video sequence of successive images of a three-dimensional (3D) scene at least two of the images being taken from respectively different camera positions, the method comprising the steps of:
-
selecting a group of images including the at least two images taken from different camera positions;
calculating a set of two-dimensional (2D) view geometry constraints for the group of images;
testing the 2D view geometry constraints to determine if the imaged scene exhibits significant 3D characteristics;
if the imaged scene exhibits significant 3D characteristics, calculating a set of 3D shape constraints for the group of images, wherein the 3D shape constraints are constrained by the 2D view geometry constraints; and
if the imaged scene exhibits significant 3D characteristics, identifying areas in the group of images that are inconsistent with 3D constraints as areas corresponding to the moving objects. - View Dependent Claims (2, 3, 4, 5, 6, 7)
selecting a plurality of target point correspondences among the successive images;
calculating respective epipolar geometries for each of the plurality of target point correspondences;
calculating a median error for each calculated epipolar geometry; and
selecting one of the epipolar geometries having a minimum median error as the 2D view geometry of the scene.
-
-
6. A method according to claim 5, further including the steps of:
-
comparing the point correspondences to the selected epipolar geometry to separate the point correspondences into inlier point correspondences that conform to the selected epipolar geometry and outlier point correspondences that do not conform to the selected epipolar geometry;
calculating a refined epipolar geometry responsive to the inlier point correspondences;
testing the refined epipolar geometry against the sequence of images to determine if the refined epipolar geometry corresponds to the dominant plane of the images or to ones of the moving objects and, if the epipolar geometry corresponds to the dominant plane, providing the epipolar geometry as the 2D view geometry of the scene;
if the refined epipolar geometry is determined to correspond to the ones of the moving objects, redefining the inlier point correspondences as further outlier point correspondences and the outlier point correspondences as further inlier point correspondences, calculating a further refined epipolar geometry responsive to the further inlier point correspondences and providing the further refined epipolar geometry as the 2D view geometry of the scene.
-
-
7. A method according to claim 6, further including the steps of:
-
selecting a first pair of images of the sequence of images;
estimating a parallax geometry for the first pair of images by selecting an initial estimate of the parallax geometry and adjusting the estimated parallax geometry to iteratively minimize errors in a parametric alignment of the first pair of images;
selecting a second pair of images of the sequence of images; and
estimating a further parallax geometry for the second pair of images by further adjusting the estimated parallax geometry to minimize errors in a parametric alignment of the second pair of images.
-
-
8. Apparatus for detecting independently moving objects in a video sequence of successive images of a three-dimensional (3D) scene at least two of the images being taken from respectively different camera positions, the apparatus comprising:
-
a video camera and a memory which are controlled to obtain a group of images of the sequence of images, the group of images including the at least two images taken from different camera positions;
means for calculating a set of two-dimensional (2D) view geometry constraints for the group of images;
means for testing the 2D view geometry constraints to determine if the imaged scene exhibits significant 3D characteristics;
means for calculating a set of 3D shape constraints for the group of images if the imaged scene exhibits significant 3D characteristics, wherein the 3D shape constraints are constrained by the 2D view geometry constraints; and
means for identifying areas in the group of images that are inconsistent with 3D constraints as areas corresponding to the moving objects if the imaged scene exhibits significant 3D characteristics. - View Dependent Claims (9, 10, 11, 12, 13, 14, 15)
means for selecting a plurality of target point correspondences from among the successive images;
means for calculating respective epipolar geometries for each of the plurality of target point correspondences;
means for calculating a median error for each calculated epipolar geometry; and
means for selecting one of the epipolar geometries having the minimum median error as the 2D view geometry of the scene.
-
-
14. Apparatus according to claim 13, further including:
-
means for comparing the point correspondences to the selected epipolar geometry to separate the point correspondences into inlier point correspondences that conform to the selected epipolar geometry and outlier point correspondences that do not conform to the selected epipolar geometry;
means for calculating a refined epipolar geometry responsive to the inlier point correspondences;
means for testing the refined epipolar geometry against the sequence of images including;
means for determining if the refined epipolar geometry corresponds to the dominant plane of the images or to ones of the moving objects; and
means for providing the refined epipolar geometry as the 2D view geometry of the scene if the refined epipolar geometry corresponds to the dominant plane;
means for redefining the inlier point correspondences as further outlier point correspondences and the outlier point correspondences as further inlier point correspondences if the refined epipolar geometry is determined to correspond to the ones of the moving objects, and for calculating a further refined epipolar geometry responsive to the further inlier point correspondences and for providing the further refined epipolar geometry as the 2D view geometry of the scene.
-
-
15. Apparatus according to claim 14, further including:
-
means for selecting a first pair of images of the sequence of images;
means for estimating a parallax geometry for the first pair of images by selecting an initial estimate of the parallax geometry and adjusting the estimated parallax geometry to iteratively minimize errors in a parametric alignment of the first pair of images;
means for selecting a second pair of images of the sequence of images; and
means for estimating a further parallax geometry for the second pair of images by further adjusting the estimated parallax geometry to minimize errors in a parametric alignment of the second pair of images.
-
-
16. An article of manufacture comprising a carrier including a plurality of computer program instructions, the computer program instructions causing a general purpose computer to perform a method of detecting independently moving objects in a video sequence of successive images of a three-dimensional (3D) scene, at least two of the images being taken from respectively different camera positions, the method comprising the steps of:
-
selecting a group of images including the at least two images taken from different camera positions;
calculating a set of two-dimensional (2D) view geometry constraints for the group of images;
testing the 2D view geometry constraints to determine if the imaged scene exhibits significant 3D characteristics;
if the imaged scene exhibits significant 3D characteristics, calculating a set of 3D shape constraints for the group of images, wherein the 3D shape constraints are constrained by the 2D view geometry constraints; and
if the imaged scene exhibits significant 3D characteristics, identifying areas in the group of images that are inconsistent with 3D constraints as areas corresponding to the moving objects. - View Dependent Claims (17, 18, 19, 20, 21)
selecting a plurality of target point correspondences among the successive images;
calculating respective epipolar geometries for each of the plurality of target point correspondences;
calculating a median error for each calculated epipolar geometries; and
selecting one of the epipolar geometries having the minimum median error as the 2D view geometry of the scene.
-
-
20. An article of manufacture according to claim 19 further including computer program instructions that cause the computer to perform the steps of:
-
comparing the point correspondences to the selected epipolar geometry to separate the point correspondences into inlier point correspondences that conform to the selected epipolar geometry and outlier point correspondences that do not conform to the selected epipolar geometry;
calculating a refined epipolar geometry responsive to the inlier point correspondences;
testing the refined epipolar geometry against the sequence of images to determine if the refined epipolar geometry corresponds to the dominant plane of the images or to ones of the moving objects and if the epipolar geometry corresponds to the dominant plane, providing the epipolar geometry as the 2D view geometry of the scene;
if the refined epipolar geometry is determined to correspond to the ones of the moving objects, redefining the inlier point correspondences as further outlier point correspondences and the outlier point correspondences as further inlier point correspondences, calculating a further refined epipolar geometry responsive to the further inlier point correspondences and providing the further refined epipolar geometry as the 2D view geometry of the scene.
-
-
21. An article of manufacture according to claim 20, further including computer program instructions that cause the computer to perform the steps of:
-
selecting a first pair of images of the sequence of images;
estimating a parallax geometry for the first pair of images by selecting an initial estimate of the parallax geometry and adjusting the estimated parallax geometry to iteratively minimize errors in a parametric alignment of the first pair of images;
selecting a second pair of images of the sequence of images; and
estimating a further parallax geometry for the second pair of images by all further adjusting the estimated parallax geometry to minimize errors in a parametric alignment of the second pair of images.
-
Specification