Method and system for 3D reconstruction of multiple views with altering search path and occlusion modeling

• US 6,856,314 B2
• Filed: 04/18/2002
• Issued: 02/15/2005
• Est. Priority Date: 04/18/2002
• Status: Active Grant

• Current Assignee: STMicroelectronics Incorporated (STMicroelectronics NV)
• Original Assignee: STMicroelectronics Incorporated (STMicroelectronics NV)
• Inventors: Ng, Kim Chai
• Primary Examiner(s): Nguyen, Kimbinh T.
• Application Number:
  US10/125,004
• Publication Number:
  US 20030198377A1
• Time in Patent Office: 1,034 Days
• Field of Search: 345/419, 345/420, 345/422, 345/421, 382/128
• US Class Current: 345/421
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06V 10/10 Image acquisition document image scanning and transmission H04N1/00; control of digital cameras H04N23/60
• CPC Class Codes: G06V 2201/12 Acquisition of 3D measurements of objects

Abstract

An image processing system and method reconstructs 3D image information corresponding to a scene from a plurality of 2D images of the scene. The method receives a plurality of image features corresponded between different 2D views of the scene, the corresponded image features deviating between different views as a result of camera relative motion. The method determines image features of the received plurality of image features that are occluded views, determines image features of the received plurality of image features that are confident seeds associated with 3D depth information, and propagates 3D depth information from the confident seeds to neighboring image features, while avoiding image features that have been determined to be occluded views. The 3D image information can be rendered and displayed such as for a virtual walkthrough of the scene.

Claims

1. A method of reconstructing 3D image information from a plurality of 2D images of a scene, the method comprising the steps of:

receiving a plurality of image features corresponded between different 2D views of a scene, the corresponded image features deviating between different views as a result of camera relative motion;

determining image features of the received plurality of image features that are occluded views;

determining image features of the received plurality of image features that are confident seeds associated with 3D depth information, the confident seeds being associated with reliable depth information; and

propagating 3D depth information from the confident seeds to neighboring image features, while avoiding image features that have been determined to be occluded views, to reconstruct 3D image information from a plurality of 2D images scene.

2. The method of claim 1, wherein the propagating step comprises the steps of:

locating confident seeds that are visible about high contrast regions in the scene; and

growing surfaces of volumes of objects from the confident seeds that are visible about high contrast regions to image features that are visible about low contrast regions.

3. The method of claim 1, wherein the propagating step comprises the steps of:

locating confident seeds that are visible about edges of objects in the scene; and

growing surfaces of volumes of objects from the confident seeds that are visible about the edges of objects to image features that are visible about homogeneous regions in the scene.

4. The method of claim 1, wherein each of the plurality of image features corresponded between different 2D views of the scene being associated with a matching curve, and wherein the step of determining image features of the received plurality of image features that are confident seeds comprises the step of:

finding the confident seeds by locating the characteristic-pair (one local maximum vs. one local minimum) of 3D depth values along the matching curve across the multiple views to determine the confident seed that has local minimum regions overlapping across a plurality of corresponding image features in the plurality of views.

5. The method of claim 1, wherein the step of propagating 3D depth information from the confident seeds to neighboring image features comprises the step of:

propagating 3D depth information from the confident seeds to neighboring image features as long as a confident seed is determined to be within the neighboring image feature'"'"'s volume of uncertainty, or error region.

7. The method of claim 1, wherein the plurality of image features corresponded between the different 2D views of the scene are pixels visible in the different 2D views.

8. The method of claim 1, wherein the 3D depth information for the image features represents locations of voxels associated with the image features.

6. A method of reconstructing 3D image information from a plurality of 2D images of a scene, the method comprising the steps of:

receiving a plurality of image features corresponded between different 2D views of a scene, the corresponded image features deviating between different views as a result of camera relative motion;

determining image features of the received plurality of image features that are occluded views;

determining image features of the received plurality of image features that are confident seeds associated with 3D depth information; and

propagating 3D depth information from the confident seeds to neighboring image features while avoiding image features that have been determined to be occluded views, to reconstruct 3D image information from a plurality of 2D images scene;

wherein the step of propagating 3D depth information from the confident seeds to neighboring image features comprises the step of propagating 3D depth information from the confident seeds to neighboring image features as long as a confident seed is determined to be within the neighboring image feature'"'"'s volume of uncertainly, or error region; and

wherein 3D depth information is propagated from the confident seed to a neighboring image feature as long as the neighboring image feature'"'"'s volume of uncertainty, or error region, substantially coincides with at least one of the confident seed'"'"'s local minimum location and global minimum location.

9. A method of reconstructing 3D image information from a plurality of 2D images of a scene, the method comprising the steps of:

receiving a plurality of image features corresponded between different 2D views of a scene, the corresponded image features deviating between different views as a result of camera relative motion;

determining image features of the received plurality of image features that are occluded views;

determining image features of the received plurality of image features that are confident seeds associated with 3D depth information; and

propagating 3D depth information from the confident seeds to neighboring image features, while avoiding image features that have been determined to be occluded views, to reconstruct 3D image information from a plurality of 2D images scene;

wherein the 3D depth information being determined for confident seeds and for neighboring image features is cross-checked for consistency among all of the non-occluded views at a depth-to-be-determined.

10. The method of claim 9, wherein the consistency cross-check among the non-occluded views includes:

determining a 3D depth information for an image feature is within a local minimum region in each of a plurality of the non-occluded views.

11. The method of claim 9, wherein the consistency cross-check among the non-occluded views includes:

determining a 3D depth information for an image feature by cross-checking that the characteristics of the subject image feature'"'"'s matching curve and the characteristics of corresponding image features matching curves in other non-occluded views substantially match with each other.

12. The method of claim 9, wherein the consistency cross-check among the non-occluded views includes:

determining a 3D depth information for an image feature by cross-checking that the characteristics of the subject image feature'"'"'s matching curve and the characteristics of corresponding image features matching curves in other non-occluded views substantially match with a specified characteristics-pair.

13. The method of claim 9, wherein a cross-check among the non-occluded views includes:

determining a 3D depth information for an image feature by cross-checking that the characteristics of the subject image feature'"'"'s matching curve and the characteristics of corresponding image features matching curves in other non-occluded views substantially match with a specified characteristics-pair; and

determining that a continuity constraint is substantially met from the subject image feature to its neighboring image features among the non-occluded views.

14. An image processing system comprising:

a memory;

a controller/processor electrically coupled to the memory;

a 3D image reconstructor, electrically coupled to the controller/processor and to the memory, for;

receiving a plurality of image features corresponded between different 2D views of a scene, the corresponded image features deviating between different views as a result of camera relative motion;

determining image features of the received plurality of image features that are occluded views;

determining image features of the received plurality of image features that are confident seeds associated with 3D depth information, the confident seeds being associated with reliable depth information; and

propagating 3D depth information from the confident seeds to neighboring image features, while avoiding image features that have been determined to be occluded views.

15. The image processing system of claim 14, further comprising at least one camera interface, electrically coupled to the controller/processor, for sending image information from at least one camera to the controller/processor.

16. The system of claim 14, wherein the controller/processor, the memory, the image feature detector, and the image matching module, are implemented in at least one of an integrated circuit, a circuit supporting substrate.

17. The image processing system of claim 14, further comprising at least one display interface for displaying a 3D image corresponding to the 3D depth information of the scene.

18. A computer readable medium including computer instructions for a 3D image reconstruction computer system, the computer instructions comprising instructions for:

receiving a plurality of image features corresponded between different 2D views of a scene, the corresponded image features deviating between different views as a result of camera relative motion;

determining image features of the received plurality of image features that are occluded views;

determining image features of the received plurality of image features that are confident seeds associated with 3D depth information, the confident seeds being associated with reliable depth information; and

propagating 3D depth information from the confident seeds to neighboring image features, while avoiding image features that have been determined to be occluded views.

19. The computer readable medium of claim 18, wherein the propagating comprises:

locating confident seeds that are visible about high contrast regions in the scene; and

growing surfaces of volumes of objects from the confident seeds that are visible about high contrast regions to image features that are visible about low contrast regions.

20. The computer readable medium of claim 18, wherein the propagating comprises:

locating confident seeds that are visible about edges of objects in the scene; and

growing surfaces of volumes of objects from the confident seeds that are visible about the edges of objects to image features that are visible about homogeneous regions in the scene.

21. The computer readable medium of claim 18, wherein each of the plurality of image features corresponded between different 2D views of the scene being associated with a matching curve, and wherein the determining image features of the received plurality of image features that are confident seeds comprises:

finding the confident seeds by locating the characteristic-pair (one local maximum vs. one local minimum) of 3D depth values along the matching curve across the multiple views to determine the confident seed that has local minimum regions overlapping across a plurality of corresponding image features in the plurality of views.

22. The computer readable medium of claim 18, wherein the propagating comprises:

propagating 3D depth information from the confident seeds to neighboring image features as long as a confident seed is determined to be within the neighboring image feature'"'"'s volume of uncertainty, or error region.

24. The computer readable medium of claim 18, wherein the plurality of image features corresponded between the different 2D views of the scene are pixels visible in the different 2D views.

25. The computer readable medium of claim 18, wherein the 3D depth information for the image features represents locations of voxels associated with the image features.

23. A computer readable medium including computer instructions for a 3D image reconstruction computer system, the computer instructions comprising instructions for:

receiving a plurality of image features corresponded between different 2D views of a scene, the corresponded image features deviating between different views as a result of camera relative motion;

determining image features of the received plurality of image features that are occluded views;

determining image features of the received plurality of image features that are confident seeds associated with 3D depth information; and

propagating 3D depth information from the confident seeds to neighboring image features, while avoiding image features that have been determined to be occluded views,

wherein the propagating comprises propagating 3D depth information from the confident seeds to neighboring image features as long as a confident seed is determined to be within the neighboring image feature'"'"'s volume of uncertainty, or error region, and

wherein 3D depth information is propagated from the confident seed to a neighboring image feature as long as the neighboring image feature'"'"'s volume of uncertainty, or error region, substantially coincides with at least one of the confident seed'"'"'s local minimum location and global minimum location.

26. A computer readable medium including computer instructions for a 3D image reconstruction computer system, the computer instructions comprising instructions for:

receiving a plurality of image features corresponded between different 2D views of a scene, the corresponded image features deviating between different views as a result of camera relative motion;

determining image features of the received plurality of image features that are occluded views;

determining image features of the received plurality of image features that are confident seeds associated with 3D depth information; and

propagating 3D depth information from the confident seeds to neighboring image features, while avoiding image features that have been determined to be occluded views,

wherein the 3D depth information being determined for confident seeds and for neighboring image features is cross-checked for consistency among all of the non-occluded views at a depth-to-be-determined.

27. The computer readable medium of claim 26, wherein the consistency cross-check among the non-occluded views includes:

determining a 3D depth information for an image feature is within a local minimum region in each of a plurality of the non-occluded views.

28. The computer readable medium of claim 26, wherein the consistency cross-check among the non-occluded views includes:

determining a 3D depth information for an image feature by cross-checking that the characteristics of the subject image features matching curve and the characteristics of corresponding image features matching curves in other non-occluded views substantially match with each other.

29. The computer readable medium of claim 26, wherein the consistency cross-check among the non-occluded views includes:

determining a 3D depth information for an image feature by cross-checking that the characteristics of the subject image feature'"'"'s matching curve and the characteristics of corresponding image features matching curves in other non-occluded views substantially match with a specified characteristics-pair.

30. The computer readable medium of claim 26, wherein a cross-check among the non-occluded views includes:

determining a 3D depth information for an image feature by cross-checking that the characteristics of the subject image feature'"'"'s matching curve and the characteristics of corresponding image features matching curves in other non-occluded views substantially match with a specified characteristics-pair; and

determining that a continuity constraint is substantially met from the subject image feature to its neighboring image features among the non-occluded views.

31. An electrical device comprising:

a data memory;

a program memory including computer instructions for 3D image reconstruction; and

a controller/processor, electrically coupled to the first interface, the data memory and the program memory, for

storing image information corresponding to a plurality of image features corresponded between different 2D views of a scene, the corresponded image features deviating between different views as a result of camera relative motion;

determining image features of the stored plurality of image features that are occluded views;

determining image features of the stored plurality of image features that are confident seeds associated with 3D depth information, the confident seeds being associated with reliable depth information; and

propagating 3D depth information from the confident seeds to neighboring image features, while avoiding image features that have been determined to be occluded views.

32. The electrical device of claim 31, further comprising:

a display interface, electrically coupled to the controller/processor, for providing 3D image display information, that corresponds to the 3D depth information, to a display.

33. The electrical device of claim 32, wherein the display interface provides 3D image display information to a display for a virtual walkthrough or the scene.

34. The electrical device of claim 31, wherein the electrical device comprises at least one of an integrated circuit and a circuit supporting substrate.

35. The electrical device of claim 31, further comprising at least one camera interface for receiving image information from at least one camera.