THREE-DIMENSIONAL INFORMATION PROCESSING APPARATUS AND METHOD

• US 20010014171A1
• Filed: 06/30/1997
• Published: 08/16/2001
• Est. Priority Date: 07/01/1996
• Status: Active Grant

• Current Assignee: Canon Ayutthaya Limited (Canon Inc.)
• Original Assignee: Canon Ayutthaya Limited (Canon Inc.)
• Inventors: YANO, KOTARO, MORI, KATSUHIKO, MATSUGU, MASAKAZU, IIJIMA, KATSUMI, KATAYAMA, TATSUSHI, SEKINE, MASAYOSHI, ISHIKAWA, MOTOHIRO, OKAUCHI, SHIGEKI, KURAHASHI, SUNAO
• Granted Patent: US 6,445,814 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/207 using a single 2D image sensor
• CPC Class Codes: H04N 13/221 using the relative movement between cameras and objects
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A three-dimensional information processing apparatus for obtaining three-dimensional information from an object having a three-dimensional shape, and performing predetermined information processing, comprises: a camera for sensing images of the object from a plurality of coordinate positions using an image sensing system having one or a plurality of optical systems. A plurality of depth information are extracted from image sensing related information sensed by the camera at the plurality of coordinate positions, and the plurality of extracted depth information are converted and unified into depth information expressed by a unified coordinate system.

Claims

1. A three-dimensional information processing apparatus for obtaining three-dimensional information from an object having a three-dimensional shape, and performing predetermined information processing, comprising:

image sensing means for sensing images of the object from a plurality of coordinate positions using an image sensing system having one or a plurality of optical systems;

information extraction means for extracting a plurality of depth information from image sensing related information obtained by said image sensing means at the plurality of coordinate positions; and

conversion/unification means for converting and unifying the plurality of depth information extracted by said depth information extraction means into depth information expressed by a unified coordinate system.

2. The apparatus according to

claim 1

, wherein said conversion/unification means obtains image information of the object from said image sensing means, detects a displacement between coordinate systems of the plurality of depth information on the basis of the obtained image information, and converts and unifies the plurality of depth information onto the unified coordinate system.

3. The apparatus according to

claim 1

, wherein the unified coordinate system has five different projection planes.

4. The apparatus according to

claim 2

, wherein the image information includes luminance information of the object, and said conversion/unification means detects the displacement between the coordinate systems on the basis of the luminance information.

5. A three-dimensional information processing method for obtaining three-dimensional information from an object having a three-dimensional shape, and performing predetermined information processing, comprising:

the first step of sensing images of the object from a plurality of coordinate positions using an image sensing system having one or a plurality of optical systems;

the second step of extracting a plurality of depth information from image sensing related information sensed at the plurality of coordinate positions in the first step; and

the third step of converting and unifying the plurality of depth information extracted by said depth information extraction means into depth information expressed by a unified coordinate system.

6. The method according to

claim 5

, wherein the third step has the step of obtaining image information of the object obtained in the first step, detecting a displacement between coordinate systems of the plurality of depth information on the basis of the obtained image information, and converting and unifying the plurality of depth information onto the unified coordinate system.

7. The method according to

claim 5

, wherein the unified coordinate system has five different protection planes.

8. The method according to

claim 6

, wherein the image information includes luminance information of the object, and the displacement between the coordinate systems is detected on the basis of the luminance information.

9. A three-dimensional information processing apparatus for obtaining three-dimensional information from an object having a three-dimensional shape, and performing predetermined information processing, comprising:

image sensing means for sensing images of the object using an image sensing system having one or a plurality of optical systems;

three-dimensional shape extraction means for extracting three-dimensional shape information of the object from image sensing related information obtained by said image sensing means; and

reliability determination means for determining reliability of the three-dimensional shape information extracted by said three-dimensional shape extraction means.

10. The apparatus according to

claim 9

, wherein said reliability determination means determines the reliability of the three-dimensional shape information on the basis of an angle of the object with respect to an image sensing plane.

11. The apparatus according to

claim 9

, wherein said reliability determination means determines the reliability of the three-dimensional shape information on the basis of a distance between said image sensing means and the object.

12. The apparatus according to

claim 9

, wherein said reliability determination means determines the reliability of the three-dimensional shape information on the basis of an angle a pad that places the object thereon makes with an image sensing plane of said image sensing means.

13. The apparatus according to

claim 9

, wherein said reliability determination means determines the reliability of the three-dimensional shape information on the basis of an area ratio of a pad that places the object thereon to an image sensing region.

14. The apparatus according to

claim 9

, wherein said reliability determination means determines the reliability of the three-dimensional shape information on the basis of a position of a pad that places the object thereon.

15. The apparatus according to

claim 9

, wherein said reliability determination means determines the reliability of the three-dimensional shape information on the basis of reflected light information reflected by the object.

16. The apparatus according to

claim 9

, wherein said reliability determination means determines the reliability of the three-dimensional shape information on the basis of a degree of correspondence of pixels between a plurality of image sensing related data sensed by said image sensing means.

17. A three-dimensional information processing apparatus for obtaining three-dimensional information from an object having a three-dimensional shape, and performing predetermined information processing, comprising:

image sensing means for sensing images of the object using an image sensing system having one or a plurality of optical systems;

three-dimensional shape extraction means for extracting three-dimensional shape information of the object from image sensing related information sensed by said image sensing means;

reliability determination means for determining reliability of the three-dimensional shape information extracted by said three-dimensional shape extraction means; and

informing means for informing a reliability determination result of said reliability determination means.

18. The apparatus according to

claim 17

, wherein said reliability determination means determines the reliability of the three-dimensional shape information on the basis of an angle of the object with respect to an image sensing plane.

19. The apparatus according to

claim 17

, wherein said reliability determination means determines the reliability of the three-dimensional shape information on the basis of a distance between said image sensing means and the object.

20. The apparatus according to

claim 17

, wherein said reliability determination means determines the reliability of the three-dimensional shape information on the basis of an angle a pad that places the object thereon makes with an image sensing plane of said image sensing means.

21. The apparatus according to

claim 17

, wherein said reliability determination means determines the reliability of the three-dimensional shape information on the basis of an area ratio of a pad that places the object thereon to an image sensing region.

22. The apparatus according to

claim 17

, wherein said reliability determination means determines the reliability of the three-dimensional shape information on the basis of a position of a pad that places the object thereon.

23. The apparatus according to

claim 17

, wherein said reliability determination means determines the reliability of the three-dimensional shape information on the basis of reflected light information reflected by the object.

24. The apparatus according to

claim 17

, wherein said reliability determination means determines the reliability of the three-dimensional shape information on the basis of a degree of correspondence of pixels between a plurality of image sensing related data sensed by said image sensing means.

25. A three-dimensional information processing apparatus for obtaining three-dimensional information from an object having a three-dimensional shape, and performing predetermined information processing, comprising:

image sensing means for sensing images of the object using an image sensing system having one or a plurality of optical systems;

three-dimensional shape extraction means for extracting three-dimensional shape information of the object from image sensing related information sensed by said image sensing means;

reliability determination means for determining reliability of the three-dimensional shape information extracted by said three-dimensional shape extraction means; and

display means for processing the three-dimensional shape information in accordance with a reliability determination result of said reliability determination means, and displaying the processed three-dimensional shape information.

26. The apparatus according to

claim 25

, wherein said reliability determination means determines the reliability of the three-dimensional shape information on the basis of an angle of the object with respect to an image sensing plane.

27. The apparatus according to

claim 25

, wherein said reliability determination means determines the reliability of the three-dimensional shape information on the basis of a distance between said image sensing means and the object.

28. The apparatus according to

claim 25

, wherein said reliability determination means determines the reliability of the three-dimensional shape information on the basis of an angle a pad that places the object thereon makes with an image sensing plane of said image sensing means.

29. The apparatus according to

claim 25

, wherein said reliability determination means determines the reliability of the three-dimensional shape information on the basis of an area ratio of a pad that places the object thereon to an image sensing region.

30. The apparatus according to

claim 25

, wherein said reliability determination means determines the reliability of the three-dimensional shape information on the basis of a position of a pad that places the object thereon.

31. The apparatus according to

claim 25

, wherein said reliability determination means determines the reliability of the three-dimensional shape information on the basis of reflected light information reflected by the object.

32. The apparatus according to

claim 25

, wherein said reliability determination means determines the reliability of the three-dimensional shape information on the basis of a degree of correspondence of pixels between a plurality of image sensing related data sensed by said image sensing means.

33. An image sensing method comprising:

the image sensing step of sensing images of an object;

the storage step of storing image information of the object;

the image sensing condition detection step of detecting a relative relationship between the object and an image sensing apparatus main body; and

the control step of controlling a storage operation of the image information,

wherein the control step includes the step of controlling the storage operation in the storage step in accordance with a detection result of the image sensing condition detection step.

34. The method according to

claim 33

, wherein the control step includes the step of controlling to store information associated with the relative relationship between the object and the image sensing apparatus main body together with sensed images sensed in the image sensing step in the storage step.

35. The method according to

claim 33

, wherein the image sensing condition detection step includes the step of detecting the relative relationship using a sensor for detecting an angle and translation movement of the image sensing apparatus main body.

36. The method according to

claim 33

, wherein the image sensing condition detection step includes the step of analyzing an object image and images around the object sensed by the image sensing apparatus main body, and detecting an angle and translation movement of the image sensing apparatus main body on the basis of changes in state of sensed images sensed in the image sensing step.

37. The method according to

claim 33

, wherein the image sensing condition detection step includes the step of analyzing an object image and images around the object sensed by the image sensing apparatus main body, and detecting changes in relative position relationship between the object and the image sensing apparatus main body on the basis of an error signal generated upon analyzing the images.

38. The method according to

claim 33

, wherein the image sensing condition detection step includes the step of analyzing an object image sensed by the image sensing apparatus main body, and detecting changes in occlusion state of the object.

39. The method according to

claim 33

, wherein the image sensing condition detection step includes the step of analyzing an object image sensed by the image sensing apparatus main body, and detecting an overlapping region area between time-serial object images.

40. The method according to

claim 33

, wherein the image sensing condition detection step includes the step of analyzing an object image sensed by the image sensing apparatus main body, and detecting changes in distance image of the object.

67. The method according to

claim 33

, wherein the image analysis step includes the step of performing an analysis calculation for acquiring a three-dimensional shape and a surface image of the object using a plurality of images.

41. An image sensing method comprising:

the image sensing step of sensing images of an object;

the analysis step of analyzing image information obtained in the image sensing step;

the image sensing condition detection step of detecting a relative relationship between the object and an image sensing apparatus main body; and

the control step of controlling an image analysis operation in the analysis step,

wherein the control step includes the step of controlling the image analysis operation in accordance with a detection result of the image sensing condition detection step.

42. The method according to

claim 41

, wherein the analysis step includes the step of performing an analysis calculation for acquiring a three-dimensional shape and a surface image of the object using a plurality of images.

43. The method according to

claim 41

, wherein the image sensing condition detection step uses a sensor for detecting an angle and translation movement of the image sensing apparatus main body.

44. The method according to

claim 41

, wherein the image sensing condition detection step includes the step of analyzing an object image and images around the object sensed by the image sensing apparatus main body, and detecting an angle and translation movement of the image sensing apparatus main body on the basis of changes in state of sensed images sensed in the image sensing step.

45. The method according to

claim 41

, wherein the image sensing condition detection step includes the step of analyzing an object image and images around the object sensed by the image sensing apparatus main body, and detecting changes in relative position relationship between the object and the image sensing apparatus main body on the basis of an error signal generated upon analyzing the images.

46. The method according to

claim 41

, wherein the image sensing condition detection step includes the step of analyzing an object image sensed by the image sensing apparatus main body, and detecting changes in occlusion state of the object.

47. The method according to

claim 41

, wherein the image sensing condition detection step includes the step of analyzing an object image sensed by the image sensing apparatus main body, and detecting an overlapping region area between time-serial object images.

48. The method according to

claim 41

, wherein the image sensing condition detection step includes the step of analyzing an object image sensed by the image sensing apparatus main body, and detecting changes in distance image of the object.

49. The method according to

claim 41

, wherein the image sensing condition detection step includes the step of stopping the image sensing step and the analysis step during a period in which neither storage processing nor analysis processing are performed.

50. An image sensing apparatus comprising:

image sensing means for sensing images of an object;

storage means for storing image information of the object;

image sensing condition detection means for detecting a relative relationship between the object and an image sensing apparatus main body; and

control means for controlling said storage means,

wherein the control means controls said storage means in accordance with an output from said image sensing condition detection means.

51. The apparatus according to

claim 50

, wherein said control means controls said storage means to store information associated with the relative relationship between the object and the image sensing apparatus main body together with sensed images sensed by said image sensing means.

52. The apparatus according to

claim 50

, wherein said image sensing condition detection means comprises a sensor for detecting an angle and translation movement of the image sensing apparatus main body.

53. The apparatus according to

claim 50

, wherein said image sensing condition detection means analyzes an object image and images around the object sensed by the image sensing apparatus main body, and detects an angle and translation movement of the image sensing apparatus main body on the basis of changes in state of sensed images sensed by said image sensing means.

54. The apparatus according to

claim 50

, wherein said image sensing condition detection means analyzes an object image and images around the object sensed by the image sensing apparatus main body, and detects changes in relative position relationship between the object and the image sensing apparatus main body on the basis of an error signal generated upon analyzing the images.

55. The apparatus according to

claim 50

, wherein said image sensing condition detection means analyzes an object image sensed by the image sensing apparatus main body, and detects changes in occlusion state of the object.

56. The apparatus according to

claim 50

, wherein said image sensing condition detection means analyzes an object image sensed by the image sensing apparatus main body, and detects an overlapping region area between time-serial object images.

57. The apparatus according to

claim 50

, wherein said image sensing condition detection means analyzes an object image sensed by the image sensing apparatus main body, and detects changes in distance image of the object.

58. An image sensing apparatus comprising:

image sensing means for sensing images of an object;

image analysis means for analyzing image information sensed by said image sensing means;

image sensing condition detection means for detecting a relative relationship between the object and an image sensing apparatus main body; and

control means for controlling said image analysis means,

wherein said control means controls said image analysis means in accordance with an output from said image sensing condition detection means.

59. The apparatus according to

claim 58

, wherein said image analysis means performs an analysis calculation for acquiring a three-dimensional shape and a surface image of the object using a plurality of images.

60. The apparatus according to

claim 58

, wherein said image sensing condition detection means comprises a sensor for detecting an angle and translation movement of the image sensing apparatus main body.

61. The apparatus according to

claim 58

, wherein said image sensing condition detection means analyzes an object image and images around the object sensed by the image sensing apparatus main body, and detects an angle and translation movement of the image sensing apparatus main body on the basis of changes in state of sensed images sensed by said image sensing means.

62. The apparatus according to

claim 58

, wherein said image sensing condition detection means analyzes an object image and images around the object sensed by the image sensing apparatus main body, and detects changes in relative position relationship between the object and the image sensing apparatus main body on the basis of an error signal generated upon analyzing the images.

63. The apparatus according to

claim 58

, wherein said image sensing condition detection means analyzes an object image sensed by the image sensing apparatus main body, and detects changes in occlusion state of the object.

64. The apparatus according to

claim 58

, wherein said image sensing condition detection means analyzes an object image sensed by the image sensing apparatus main body, and detects an overlapping region area between time-serial object images.

65. The apparatus according to

claim 58

, wherein said image sensing condition detection means analyzes an object image sensed by the image sensing apparatus main body, and detects changes in distance image of the object.

66. The apparatus according to

claim 58

, wherein said image sensing condition detection means stops operations of said image sensing means and said image analysis means during a period in which neither storage processing nor analysis processing are performed.