Three-dimensional information acquisition apparatus and three-dimensional information acquisition method

• US 7,092,563 B2
• Filed: 06/24/2002
• Issued: 08/15/2006
• Est. Priority Date: 06/26/2001
• Status: Expired due to Fees

• Current Assignee: Olympus Optical Company Ltd. (Olympus Corporation)
• Original Assignee: Olympus Optical Company Ltd. (Olympus Corporation)
• Inventors: Shiratani, Fumiyuki
• Primary Examiner(s): Bella, Matthew C.
• Assistant Examiner(s): DESIRE, GREGORY M
• Application Number:
  US10/179,494
• Publication Number:
  US 20020196415A1
• Time in Patent Office: 1,513 Days
• Field of Search: 345/419, 345/420, 356/603, 356/623, 356/610, 382/154, 382/195, 382/203, 382/106, 353/31
• US Class Current: 382/154
• CPC Class Codes: G01B 11/25 by projecting a pattern, e.g. one or more lines, moiré fringes on the object G01B11/255 takes precedence ; image analysis for depth or shape recovery G06T7/50
• CPC Class Codes: G02B 7/28 Systems for automatic generation of focusing signals
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/207 using a single 2D image sensor
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 5/2226 Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10

Abstract

There are performed three types of object shooting, i.e., a projection pattern shooting using a pattern including both characterization of a stripe based on a color and characterization based on a gradation of the same component, a flash shooting without a pattern and a non-flash shooting without a pattern. A two-dimensional image information generation portion corrects the influence of a surface reflectivity of an object or external light from the three types of images, and estimates a pattern structure using the characteristic based on a color with respect to an object having a white-based color or a low-saturation color and based on a gradation with respect to an object having a high-saturation color. A depth calculation portion specifies the correspondence relationship between the estimated pattern structure and information of the projected pattern, and calculates a depth of each part of the object based on a result of specification.

Claims

1. A three-dimensional information acquisition apparatus which shoots, with a shooting unit, a shooting object on which a predetermined projection pattern, in which patterns of a plurality of color components overlap, is projected at a position a predetermined distance away from the shooting unit, and which analyses an image shot by said shooting unit, thereby obtaining three-dimensional information of said shooting object, said apparatus comprising:

an image memory which receives from said shooting unit and stores therein;

a first image obtained by shooting said shooting object illuminated with light from a light source, a second image obtained by shooting said shooting object without performing illumination with the light from said light source by one of turning off said light source and preventing transmission of the light from said light source, and a third image obtained by shooting said shooting object on which said projection pattern is projected by utilizing one of the light from said light source and light from any other light source having a same characteristic as a characteristic of said light source;

a pattern memory which stores therein said projection pattern to be projected onto said shooting object in advance;

a pattern structure estimation portion which corrects influence of a surface reflectivity of said shooting object and external light by utilizinq said first, second and third images stored in said image memory, and which estimates a structure of said shot pattern based on at least one of a color and a gradation of each portion of said pattern obtained by performing said correction; and

a depth calculation portion which specifies a correspondence relationship between a result of said estimation obtained by said pattern structure estimation portion and said projection pattern stored in said pattern memory, and which calculates a depth value of said shooting object based on a result of specification of said correspondence relationship.

2. An apparatus according to claim 1, wherein a pattern of each color component in said projection pattern is obtained by arranging a plurality of areas in such a manner that adjacent areas have different gradations.

3. An apparatus according to claim 1, wherein a pattern of each color component in said projection pattern is obtained by alternately arranging an area where a gradation becomes a local maximum and an area where a gradation becomes a local minimum.

4. An apparatus according to claim 3, wherein said area where a gradation becomes a local minimum is on a minimum gradation level.

5. An apparatus according to claim 1, wherein said projection pattern includes a plurality of areas having different chromatic colors, and

wherein said areas having chromatic colors in said projection pattern comprise combinations of areas each having a predetermined gradation of a pattern of said each color component, wherein a gradation of an arbitrary area in a pattern of one color component is different from a gradation of an arbitrary area in a pattern of another color component, and wherein said projection pattern is thereby encoded so as to be capable of specifying an area in said projection pattern to which said arbitrary area corresponds.

6. An apparatus according to claim 1, wherein a plurality of said color components are three components of R, G and B.

7. An apparatus according to claim 1, wherein said pattern structure estimation portion selectively executes a first estimation method which estimates a structure of said shot pattern based on a color of said shot pattern, and a second estimation method which estimates a structure of said shot pattern based on a gradation of at least one color component which is not all the color components, and

wherein said depth calculation portion calculates a depth value of said shooting object by utilizing a result of estimation obtained by one of said first and second estimation methods.

8. A three-dimensional information acquisition apparatus which shoots, with a shooting unit, a shooting object on which a predetermined projection pattern, in which patterns of a plurality of color components overlap, is projected at a position a predetermined distance away from the shooting unit, and which analyses an image shot by said shooting unit, thereby obtaining three-dimensional information of said shooting object, said apparatus comprising:

an image memory which receives from said shooting unit and stores therein;

a first image obtained by shooting said shooting object illuminated with light from a light source and a second image obtained by shooting said shooting object on which said projection pattern is projected by utilizing one of the light from said light source and light from any other light source having a same characteristic as a characteristic of said light source;

a pattern memory which stores said projection pattern projected on said shooting object in advance;

a pattern structure estimation portion which corrects an influence of a surface reflectivity of said shooting object by utilizing said first and second images stored in said image memory, and which estimates a structure of a shot pattern based on at least one of a color and a gradation of each portion of said pattern obtained by performing said correction; and

a depth calculation portion which specifies a correspondence relationship between a result of said estimation obtained by said pattern structure estimation portion and said projection pattern stored in said pattern memory, and which calculates a depth value of said shooting object by utilizing a result of specification of said correspondence relationship.

9. An apparatus according to claim 8, wherein a pattern of each color component in said projection pattern is obtained by arranging a plurality of areas in such a manner that adjacent areas have different gradations.

10. An apparatus according to claim 8, wherein a pattern of each color component in said projection pattern is obtained by alternately arranging an area where a gradation becomes a local maximum and an area where a gradation becomes a local minimum.

11. An apparatus according to claim 10, wherein said area where a gradation becomes a local minimum is on a minimum gradation level.

12. An apparatus according to claim 8, wherein said projection pattern includes a plurality of areas having different chromatic colors, and

wherein said areas having chromatic colors in said projection patterns comprise combinations of areas each having a predetermined gradation of a pattern of said each color component, wherein a gradation of an arbitrary area in a pattern of one color component is different from a gradation of an arbitrary area in a pattern of another color component, and wherein said projection pattern is thereby encoded so as to be capable of specifying an area in said projection pattern to which said arbitrary area corresponds.

13. An apparatus according to claim 8, wherein a plurality of said color components are three components of R, G and B.

14. An apparatus according to claim 8, wherein said pattern structure estimation portion selectively executes a first estimation method which estimates a structure of said shot pattern based on a color of said shot pattern, and a second estimation method which estimates a structure of said shot pattern based on a gradation of at least one color component which is not all the color components, and

wherein said depth calculation portion calculates a depth value of said shooting object by utilizing a result of estimation obtained by one of said first and second estimation methods.

15. A three-dimensional information acquisition apparatus which shoots, with a shooting unit, a shooting object on which a predetermined projection pattern, in which patterns of a plurality of color components overlap, is projected at a position a predetermined distance away from the shooting unit, and which analyses an image shot by said shooting unit, thereby obtaining three-dimensional information of said shooting object, said apparatus comprising:

an image memory which receives and stores therein an image obtained by shooting said shooting object on which said projection pattern is projected from said shooting unit;

a pattern memory which stores therein said projection pattern to be projected on said shooting object in advance;

a pattern structure estimation portion which selectively executes a first estimation method which estimates a structure of said shot pattern based on a color of said shot pattern included in said image, and a second estimation method which estimates a structure of said shot pattern based on a gradation of at least one color component which is not all color components; and

a depth calculation portion which specifies a correspondence relationship between a result of estimation obtained by one of said first and second estimation methods in said pattern structure estimation portion and said projection pattern stored in said pattern memory, and which calculates a depth value of said shooting object by utilizing a result of specifying said correspondence relationship.

16. An apparatus according to claim 15, wherein a pattern of each color component in said projection pattern is obtained by arranging a plurality of areas in such a manner that adjacent areas have different gradations.

17. An apparatus according to claim 15, wherein a pattern of each color in said projection pattern is obtained by alternately arranging an area where a gradation becomes a local maximum and an area where a gradation becomes a local minimum.

18. An apparatus according to claim 17, wherein said area where a gradation becomes a local minimum is on a minimum gradation level.

19. An apparatus according to claim 15, wherein said projection pattern includes a plurality of areas having different chromatic colors, and

wherein said areas having chromatic colors in said projection pattern comprise combinations of areas each having a predetermined gradation of a pattern of said each color component, wherein a gradation of an arbitrary area in a pattern of one color component is different from a gradation of an arbitrary area in a pattern of another color component, and wherein said projection pattern is thereby encoded so as to be capable of specifying an area in said projection pattern to which said arbitrary area corresponds.

20. An apparatus according to claim 15, wherein a plurality of said color components are three components of R, G and B.

21. A three-dimensional information acquisition method wherein a shooting unit is used to shoot a shooting object on which a predetermined projection pattern, in which patterns of a plurality of color components overlap, is projected at a position a predetermined distance away from said shooting unit, and wherein an image shot by said shooting unit is analysed, thereby acquiring three-dimensional information of said shooting object, said method comprising:

obtaining a first image by shooting said shooting object illuminated with light from a light source;

obtaining a second image by shooting said shooting object without performing illumination using the light from said light source by one of turning off said light source and preventing transmission of the light from said light source;

obtaining a third image by shooting said shooting object on which said projection pattern is projected by utilizing one of the light from said light source and light from any other light source having a same characteristic as a characteristic of said light source;

correcting influence of a surface reflectivity of said shooting object and external light by utilizing said first, second and third images, and estimating a structure of a shot pattern based on at least one of a color and a gradation of each portion of said pattern obtained by performing said correction;

specifying a correspondence relationship between a result of said estimation and said projected predetermined projection pattern; and

calculating a depth value of said shooting object based on a result of specifying said correspondence relationship.

22. A three-dimensional information acquisition method wherein a shooting unit is used to shoot a shooting object on which a predetermined projection pattern, in which patterns of a plurality of color components overlap, is projected at a position a predetermined distance away from said shooting unit, and wherein an image shot by said shooting unit is analysed, thereby acquiring three-dimensional information of said shooting object, said method comprising:

obtaining a first image by shooting said shooting object illuminated with light from a light source;

obtaining a second image by shooting said shooting object on which said projection pattern is projected by utilizing one of the light from said light source and light from any other light source having a same characteristic as a characteristic of said light source;

correcting influence of a surface reflectivity of said shooting object by utilizing said first and second images, and estimating a structure of a shot pattern based on at least one of a color and a gradation of each portion in said pattern obtained by performing said correction;

specifying the a correspondence relationship between a result of said estimation and said projected predetermined projection pattern; and

calculating a depth value of said shooting object by utilizing a result of specifying said correspondence relationship.

23. A three-dimensional information acquisition method wherein a shooting unit is used to shoot a shooting object on which a predetermined projection pattern, in which patterns of a plurality of color components overlap, is projected at a position a predetermined distance away from said shooting unit, and wherein an image shot by said shooting unit is analysed, thereby acquiring three-dimensional information of said shooting object, said method comprising:

obtaining an image by shooting said shooting object on which said projection pattern is projected;

selectively executing a first estimation method which estimates a structure of a shot pattern based on a color of said shot pattern included in said image and a second estimation method which estimates a structure of said shot pattern based on a gradation of at least one color component which is not all color components;

specifying a correspondence relationship between a result of estimation by one of said first and second estimation methods and said projected predetermined projection pattern; and

calculating a depth value of said shooting object by utilizing a result of specifying said correspondence relationship.