Computer vision depth segmentation using virtual surface

• US 20030038875A1
• Filed: 09/18/2001
• Published: 02/27/2003
• Est. Priority Date: 08/17/2001
• Status: Active Grant

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Wren, Christopher R., Ivanov, Yuri A., Pentland, Alex P.
• Granted Patent: US 6,911,995 B2
• Time in Patent Office: Days
• US Class Current: 348/43
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/15 for colour aspects of image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/286 having separate monoscopic and stereoscopic modes
• CPC Class Codes: H04N 13/324 Colour aspects
• CPC Class Codes: H04N 13/363 using image projection screens volumetric display H04N13/388
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0092 Image segmentation from stereoscopic image signals

Abstract

The location of an object in a physical scene is identified with a stereo camera. A virtual surface is identified in the physical scene, and an approximate disparity set is constructed for the virtual surface. A main and a reference image of the scene are acquired by the stereo camera. The reference image is warped according to the disparity set, and subtracted from the main image to determine a set of depth residual values. Pixels having a substantially non-zero residual are identified with a surface of the object not coincident with the virtual surface.

Claims

1. A method for identifying a location of an object in a physical scene with a stereo camera comprising:

identifying a virtual surface in the physical scene;

constructing an approximate disparity set for the virtual surface;

acquiring a main and a reference image of the scene with the stereo camera;

warping the reference image according to the disparity set;

subtracting pixels of the warped reference image from corresponding pixels of the main image to determine a depth residual of each pixel; and

identifying pixels having a substantially non-zero residual with a surface of the object not coincident with the virtual surface.

2. The method of claim 1 wherein the virtual surface has an associated margin to form a virtual volume near the virtual surface with a thickness equal to the margin.

3. The method of claim 1 wherein the virtual surface is an arbitrary surface defined in a space of the physical scene.

4. The method of claim 1 wherein the virtual surface is partially tangible and partially in a space of the scene.

5. The method of claim 1 further comprising:

setting each depth residual less than a predetermined threshold to zero; and

setting all other depth residuals to one to generate a binary segmentation mask for the object.

6. The method of claim 1 wherein the object is moving, and further comprising:

tracking the moving object in a stereo video of the scene using the binary segmentation mask.

7. The method of claim 1 further comprising:

acquiring a sparse set of point correspondences from a calibration pair of images;

applying a polynomial interpolation to the sparse set of point correspondences to generate a smooth continuous approximate disparity set.

8. The method of claim 7 wherein a particular disparity, d(x, y) is approximated by a linear system d(x, y)=Λ

{tilde over (x)}(x, y), where Λ

is an unknown matrix of coefficients, and {tilde over (x)}(x, y) is a power expansion of x=[x, Y]^T

9. The method of claim 1 wherein the virtual surface is substantially planar and the approximated disparity set is obtained from intrinsic camera parameters of the stereo camera.

10. The method of claim 1 further comprising:

determining a touching of the virtual surface by the object from the depth disparities.

11. The method of claim 1 further comprising:

illuminating the scene and the object with a dynamic projector.

12. The method of claim 11 wherein the illumination includes a high contrast image.

13. The method of claim 2 further comprising:

performing volumetric depth segmentation operations according to virtual volume.

14. The method of claim 1 further comprising:

identifying a first virtual surface in the physical scene;

identifying a second virtual surface in the physical scene offset from the first virtual surface by a constant distance;

analytically constructing an approximate disparity set for the first virtual surface and the second virtual surface;

warping the reference image according to the first disparity set;

warping the reference image according to the second disparity set;

subtracting each pixel of the first warped reference image from a corresponding pixel of the main image to determine a first depth residual of each pixel; and

subtracting each pixel of the second reference image from a corresponding pixel of the main image to determine a second depth residual of each pixel; and

comparing the first and second depth residuals to determine a touching of the virtual surface.

15. A system for identifying a location of an object in a physical scene comprising:

means for identifying a virtual surface in the physical scene;

means for analytically constructing an approximate disparity set for the virtual surface;

a stereo camera acquiring a main and a reference image of the scene;

means for warping the reference image according to the disparity set;

means for subtracting each pixel of the warped reference image from a corresponding pixel of the main image to determine a depth residual of each pixel; and

means for identifying each pixel having a substantial non-zero depth residual with a surface of the object not coincident with the virtual surface.

16. The system of claim 15 wherein the virtual surface has an associated margin to form a virtual volume near the virtual surface with a thickness equal to the margin.

17. The system of claim 15 wherein the virtual surface is an arbitrary surface defined in a space of the physical scene.

18. The system of claim 15 wherein the virtual surface is partially tangible and partially in a space of the scene.

19. The system of claim 15 further comprising:

means for setting each depth residual less than a predetermined threshold to zero; and

means for setting all other depth residuals to one to generate a binary segmentation mask for the object.

20. The system of claim 15 wherein the object is moving, and further comprising:

means for tracking the moving object in a stereo video of the scene using the binary segmentation mask.