Depth information measurement apparatus and mixed reality presentation system

• US 7,295,697 B1
• Filed: 08/28/2000
• Issued: 11/13/2007
• Est. Priority Date: 12/06/1999
• Status: Expired due to Fees

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Satoh, Kiyohide
• Primary Examiner(s): Bali; Vikkram
• Application Number:
  US09/648,295
• Time in Patent Office: 2,633 Days
• Field of Search: 382/100, 382/106, 382/108, 382/154, 359/263, 359/318, 345582-588, 356/3, 356/3.01, 356/3.09
• US Class Current: 382/154
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/596 from three or more stereo images

Abstract

There is disclosed a depth information generation apparatus capable of acquiring high-precision depth information within a short computation time. A depth information generation apparatus of this invention generates depth information at the capture position of a reference image from the reference image, and at least one peripheral image that forms a stereo image pair with the reference image, and has a high-speed stereo processor (30) for generating depth information at high speed from the reference image and peripheral image, and a high-precision stereo processor (40) for generating high-precision depth information. A motion detector (20) detects a motion in an image, and instructs an image composition unit (50) to select the output from the high-speed stereo processor (30) for a portion with a large motion, and the output from the high-precision stereo processor (40) for other portions. The image composition unit (50) composites the outputs from the high-speed stereo processor (30) and high-precision stereo processor (40) in accordance with an instruction from the motion detector (20), and outputs a depth map as final depth information.

Claims

1. A depth information measurement apparatus for measuring depth information at a capture position of a reference image on the basis of the reference image, and at least one peripheral image that forms a stereo image pair with the reference image, comprising:

a plurality of depth information measurement means for measuring depth information from the reference image and peripheral image, said plurality of depth information measurement means using different measurement methods;

selection means for selecting one of a plurality of depth information for each predetermined region of the reference image; and

composition means for compositing the depth information selected by said selection means and outputting the composited information as depth information at the capture position of the reference image.

2. The apparatus according to claim 1, wherein said plurality of depth information measurement means having different measurement precisions and different measuring process speeds from each other, for measuring a plurality of depth information.

3. The apparatus according to claim 1, wherein said plurality of depth information measurement means, comprising:

first depth information measurement means for measuring first depth information from the reference image and the peripheral image;

second depth information measurement means for measuring second depth information from the reference image and the peripheral image by a method different from said first depth information measurement means.

4. The apparatus according to claim 3, wherein said first depth information measurement means has a higher processing speed than said second depth information measurement means, and the second depth information has higher precision than the first depth information.

5. The apparatus according to any one of claims 1 to 4, wherein said selection means selects in accordance with a predetermined condition.

6. The apparatus according to claim 5, wherein the predetermined condition is a condition obtained from the reference image and/or the peripheral image.

7. The apparatus according to claim 6, wherein said selection means comprises motion detection means for detecting a motion in each predetermined region from the reference image, and

said predetermined condition is based on an amount and/or a direction of the motion detected by said motion detection means.

8. The apparatus according to claim 6, wherein one of said plurality of depth information measurement means comprises reliability detection means for detecting a reliability of the measured depth information, and

said predetermined condition is based on the reliability detected said reliability detection means.

9. The apparatus according to claim 4, wherein said selection means comprises motion detection means for detecting a motion in each predetermined region from the reference image, and

said selection means selects the first depth information for the predetermined region in which the detected motion has not less than a predetermined amount, and selects the second depth information for other regions.

10. The apparatus according to claim 4, wherein said first depth information measurement means comprises reliability detection means for detecting a reliability of the first depth information, and

said selection means selects the first depth information for the predetermined region in which the detected reliability has not less than a predetermined amount, and selects the second depth information for other regions.

11. The apparatus according to claim 4, wherein said second depth information measurement means measures the second depth information using a condition which is not considered when said first depth information measurement means measures the first depth information.

12. The apparatus according to claim 11, wherein the conditions include occlusion by an object.

13. The apparatus according to any one of claims 1 to 4, wherein said depth information output from said composition means are one of disparity information, a disparity map, distance information, and a range image.

14. A mixed reality presentation system including a depth information measurement apparatus cited in any one of claims 1 to 4.

15. A storage medium which stores a program that can be executed by a computer apparatus, and makes the computer apparatus that has executed the program function as a depth information measurement apparatus cited in any one of claims 1 to 4.

20. The method according to any one of claims 1 to 4, wherein said selection step selects in accordance with a predetermined condition.

21. The method according to claim 20, wherein the predetermined condition is a condition obtained from the reference image and/or the peripheral image.

22. The method according to claim 21, wherein said selection step comprises motion detection step of detecting a motion in each predetermined region from the reference image, and

said predetermined condition is based on an amount and/or a direction of the motion detected by said motion detection step.

23. The method according to claim 21, wherein said measurement step has reliability detection step of detecting a reliability of the measured depth information, and

said predetermined condition is based on the reliability detected said reliability detection step.

28. The method according to any one of claims 1 to 4, wherein said depth information output from said composition step are one of disparity information, a disparity map, distance information, and a range image.

16. A depth information measurement method for measuring depth information at a capture position of a reference image on the basis of the reference image, and at least one peripheral image that forms a stereo image pair with the reference image, comprising steps of:

a measurement step of measurement a plurality of depth information from the reference image and peripheral image, using different measurement methods;

a selection step of selecting one of the plurality of depth information for each predetermined region of the reference image; and

a composition step of compositing the depth information selected by said selection step and outputting the composited information as depth information at the capture position of the reference image.

17. The method according to claim 16, wherein said measurement step measures the plurality of depth information at different measurement precisions and different measuring process speeds from each other.

18. The method according to claim 16, wherein said measurement step, comprising:

first depth information measurement step of measuring first depth information from the reference image and the peripheral image;

second depth information measurement step of measuring second depth information from the reference image and the peripheral image by a method different from said first depth information measurement step.

19. The method according to claim 18, wherein said first depth information measurement step utilizes a higher processing speed than that of said second depth information measurement step, and the second depth information has higher precision than that of the first depth information.

24. The method according to claim 19, wherein said selection step comprises motion detection step of detecting a motion in each predetermined region from the reference image, and

said selection step selects the first depth information for the predetermined region in which the detected motion has not less than a predetermined amount, and selects the second depth information for other regions.

25. The method according to claim 19, wherein said first depth information measurement step comprises reliability detection step of detecting a reliability of the first depth information, and

said selection step selects the first depth information for the predetermined region in which the detected reliability has not less than a predetermined amount, and selects the second depth information for other regions.

26. The method according to claim 19, wherein said second depth information measurement step measures the second depth information using a condition which is not considered when said first depth information measurement step measures the first depth information.

27. The method according to claim 26, wherein the conditions include occlusion by an object.

29. A mixed reality presentation system executing a depth information measurement method cited in any one of claims 16 to 19 and 24 to 28.

30. A storage medium which stores a program that can be executed by a computer apparatus, and makes the computer method that has executed the program function as a depth information measurement method cited in any one of claims 16 to 19 and 24 to 28.