Scanning system and calibration method for capturing precise three-dimensional information of objects

• US 7,068,825 B2
• Filed: 04/13/2001
• Issued: 06/27/2006
• Est. Priority Date: 03/08/1999
• Status: Expired due to Fees

• Current Assignee: Orametrix Incorporated
• Original Assignee: Orametrix Incorporated
• Inventors: Weise, Thomas, Imgrund, Hans, Rubbert, Rüdger, Sporbert, Peer, See, Peter, Geerdes, Hans-Florian, Pfeil, Lutz
• Primary Examiner(s): Miriam, Daniel
• Assistant Examiner(s): Choobin, Barry
• Application Number:
  US09/834,593
• Publication Number:
  US 20010038705A1
• Time in Patent Office: 1,901 Days
• Field of Search: 382/128, 382/131, 382/108, 382/106, 382/154, 382/181, 382/199, 382/201, 345/630, 345/419, 356/3.06, 356/39, 377/10, 128/922
• US Class Current: 382/128
• CPC Class Codes: A61C 7/00 Orthodontics, i.e. obtaining or maintaining the desired position of teeth, e.g. by straightening, evening, regulating, separating, or by correcting malocclusions
• CPC Class Codes: A61C 7/002 Orthodontic computer assisted systems
• CPC Class Codes: A61C 7/146 Positioning or placement of brackets; Tools therefor
• CPC Class Codes: A61C 9/006 projecting one or more stripes or patterns on the teeth
• CPC Class Codes: A61F 2002/30953 using a remote computer network, e.g. Internet
• CPC Class Codes: B33Y 50/00 Data acquisition or data processing for additive manufacturing
• CPC Class Codes: B33Y 80/00 Products made by additive manufacturing
• CPC Class Codes: G01B 11/2504 Calibration devices
• CPC Class Codes: G01B 11/2518 Projection by scanning of the object
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light

Abstract

A scanning system includes a hand-held scanning device that generates two-dimensional images of a pattern reflected off an object. The system also includes a memory and processing unit. The memory stores a calibration table for the scanner and received scanned bitmap images. The processing unit generates three-dimensional information as to a scanned object.

The scanning can be performed without knowledge or even precise control of the position of the object relative to the scanner. Random movement of the object during scanning is also possible. For example, the scanner is simply swept over the surface of the object by hand. Three-dimensional information of the object is obtained from the captured images using a calibration table for the scanner. A method of calibration of the scanner in X, Y and Z directions is also described. The scanner can be used for a variety of purposes, including medical and industrial purposes. The illustrated embodiment is in-vivo scanning of human teeth for purposes of orthodontic treatment planning and diagnosis.

Claims

1. A scanner system for capturing three-dimensional information of an object, comprising:

a projection system projecting a pattern onto said object along a first axis;

an electronic imaging device comprising an array of pixels, said electronic imaging device oriented along a second axis different from said first axis, said electronic imaging device forming an image of said pattern after reflection of said pattern off of said object;

a memory storing a three-axis (X, Y and Z) coordinate system calibration relationship for said scanner, said calibration relationship identifying;

(1) pixel coordinates for said electronic imaging device for numerous portions of said pattern, said pixel coordinates associated with distance information from said projection system in a Z direction at at least two different Z distances, and (2) distance information in X and Y directions, for said numerous portions of said pattern, at said at least two different Z distances; and

at least one processing unit processing said image of said pattern and comparing data from said image to said calibration relationship to thereby derive spatial information in three dimensions of points on said object reflecting said projected pattern onto said electronic imaging device.

2. The scanner system of claim 1, wherein said processing system derives said spatial information in three dimensions from said calibration relationship without use of optical characteristics of lens systems associated with said projection system and said electronic imaging device, to thereby compensate for optical distortions present in said lens system.

3. The scanner system of claim 1, wherein said calibration relationship (1) is obtained by projection of said pattern onto at least one surface and detecting at least a portion of the pattern with said electronic imaging device with said at least one surface placed at at least two known Z distances from said projection device.

4. The scanner system of claim 3, wherein said at least one surface comprises a reference object of known geometry.

5. The scanner system of claim 4, wherein said reference object comprises a planar surface.

6. The scanner system of claim 3, wherein said reference object comprises a curved surface.

7. The scanner system of claim 1, wherein said calibration relationship (2) is obtained, at least in part, by imaging a reference object having a plurality of clearly distinguishable objects spatially separated from each other in X and Y directions at said at least two different Z distances.

8. The scanner system of claim 7, wherein said reference object comprises a plane having an array of X-Y points.

9. The scanner system of claim 7, wherein said array of points comprises at least four points.

10. The scanner system of claim 7, wherein said array of points comprises apertures in a planar surface which are backlit by a source of illumination.

11. The scanner system of claim 7, wherein said reference object is moved between said two different Z distances to thereby generate said calibration relationship (2).

12. The scanner of claim 1, wherein said projection system and said electronic imaging device are contained in a housing sized and shaped to be held in a human hand.

13. The scanner system of claim 12, wherein said housing further comprises a distal portion holding a mirror, and wherein said distal portion is sized and shaped so as to be insertable into and moveable within an oral cavity of a human so as to enable scanning of anatomical structures inside the oral cavity.

14. The scanner system of claim 1, wherein said memory and processing unit are located in a scanning workstation, said scanning workstation further comprising a monitor operatively connected to said processing unit, said monitor displaying three-dimensional images of said object during or after said processing unit has derived said spatial information.

15. The scanner system of claim 1, wherein said projection means illuminates said object at in a series of flashes at a rate of at least one flash per second, and wherein said electronic imaging device captures a series of images of said pattern after reflection from said object at a frame rate substantially in synchronism with said rate.

16. The scanner system of claim 1, wherein said object comprises a work of art.

17. The scanner system of claim 1, wherein said object comprises a human body.

18. The scanner system of claim 1, wherein said object comprises a workpiece.

19. The scanner system of claim 1, wherein said processing unit executes software performing (1) pattern recognition of images captured by said electronic imaging device, (2) decoding of patterns detected by said pattern recognition software to identify specific portions of said projected pattern captured by said electronic imaging device;

and (3) calculation of spatial coordinates in three dimensions for said specific portions of said pattern in accordance with said calibration relationship stored in said memory.

20. The scanner system of claim 1, wherein said processing unit further executes a registration algorithm for registering three-dimensional spatial coordinates assigned to a first frame derived from a first captured image with three-dimensional spatial coordinates assigned to a second frame derived from a second captured image.

21. The scanner system of claim 1, wherein said calibration relationship comprises a calibration table, and wherein said calibration table contains pixel coordinates in calibration relationship (1) in sub-pixel resolution.

22. The scanner system of claim 21, wherein said calibration table contains said distance information with a resolution greater than 1 millimeter.

23. The scanner system of claim 1, wherein a reflective material is applied to said object prior to capturing images of said object with said electronic image converter.

24. The scanner system of claim 1, wherein said object comprises a tooth.

25. The scanner system of claim 1, wherein said projection pattern portion comprises a series of parallel lines, and wherein said projection pattern contain elements that vary in a direction both parallel to said lines and perpendicular to said lines.

80. The apparatus of claim 1 wherein said scanner further comprises source of general illumination.

26. A method of obtaining three-dimensional surface information of an object, comprising the steps of:

positioning a scanner proximate to said object, said scanner having a projection system for projecting a pattern onto said object and an electronic imaging device generating two dimensional images of the reflection of said pattern off of said surface;

moving said scanner and object relative to each other;

capturing, with said electronic imaging device, a series of two dimensional images of said projection pattern after reflection of said pattern off of said object as said scanner and said surface move relative to each other;

processing said series of images to obtain a series of frames of said object, each frame comprising spatial coordinates for a portion of the surface of said object in three dimensions, and

registration overlapping areas of said frames to each other to generate a three-dimensional virtual model of said object.

27. The method of claim 26, wherein said series of images are captured at a rate of at least one image per second and wherein said pattern is projected onto said object in a series of flashes timed at a rate in synchronism with said image capture rate.

28. The method of claim 27, wherein after initiation of said series of flashes said flashes continue automatically without human intervention.

29. The method of claim 26, wherein said step of processing further comprises a step of executing a decoding algorithm identifying specific portions of said projected pattern captured by said electronic imaging device after reflection from said object.

30. The method of claim 26, further comprising the step of storing a calibration relationship for said scanner wherein said processing system derives said spatial information in three dimensions from said calibration relationship without use of optical characteristics of lens systems associated with said projection system and said electronic imaging device, to thereby compensate for optical distortions present in said lens system.

31. The method of claim 30, wherein said calibration relationship comprises a mathematical function providing Z distance information and X and Y values from input comprising pixel addresses in said electronic imaging device.

32. The method of claim 30, wherein said calibration relationship comprises a calibration table.

33. The method of claim 26, further comprising the step of providing a data storage medium storing two dimensional images obtained by said electronic imaging device needed to cover said object.

34. The method of claim 26, wherein said step of processing comprising the steps of:

a) performing pattern recognition of said series of images to detect features of said projected pattern in each of said images,

b) decoding of patterns detected by said pattern recognition step to identify specific portions of said projected pattern captured by said electronic imaging device; and

c) calculation of spatial coordinates in three dimensions for said specific portions of said decoded patterns.

35. The method of claim 34, wherein said step of calculation comprises comparing decoded points of said projected pattern captured by said electronic imaging device with calibration values stored in a calibration table stored in a memory for said scanner.

36. The method of claim 34, wherein each two-dimensional image captured by said electronic imaging device is converted to a frame in accordance with said calculation step, and wherein said series of frames are stored in a memory, and wherein the method further comprises the step of executing a registration algorithm to register said series of frames to derive a three-dimensional model of the surface of said object.

37. The method of claim 26, wherein said scanner comprises a hand-held unit.

38. The method of claim 26, wherein said scanner is used to obtain a three-dimensional model of an anatomical structure.

39. The method of claim 38, wherein said anatomical structure comprises teeth.

40. The method of claim 39, wherein said teeth are scanned in-vivo with said scanner.

41. The method of claim 39, wherein a physical model is made from said anatomical structure and wherein said scanner scans said physical model.

42. The method of claim 41, wherein said physical model comprises a model of a patient'"'"'s teeth and surrounding anatomical structures.

81. The method of claim 26, wherein said scanner further comprises a source of general illumination.

43. A method of calibration of a scanner, said scanner comprising a projection system for projecting a pattern and an electronic imaging device for generating an image of said pattern after reflection of said pattern from an object, comprising the steps of:

projecting said pattern onto at least one surface at two different distances from said scanner and generating first and second images with said electronic imaging device of said pattern at said two distances;

imaging a set of objects of known spatial X-Y relationship at said two different differences;

generating a three-axis (X, Y and Z) coordinate system calibration relationship for said scanner, said calibration relationship indicating;

(1) pixel coordinates for numerous portions of said pattern when said pattern is projected onto said surface at said two different distances, and (2) distance information in X and Y directions for said portions of said pattern for said two different distances; and

storing said calibration relationship in a memory associated with said scanner.

44. The method of claim 43, wherein said set of objects of known spatial X-Y relationship comprises an array of points lying in a plane and separated from each other by a known distance.

45. The method of claim 44, where said step of generating said calibration relationship comprises the step of interpolating said distance information in X and Y directions for portions of said pattern from said images of said array of points.

46. The method of claim 43, wherein said pixel coordinates for said numerous portions of said pattern are obtained by performing the following steps:

a) performing pattern recognition of said first and second images to detect features of said projected pattern in each of said images, and

b) associating said specific portions of said projected pattern to specific pixels, or sub-pixels, of said electronic imaging device.

47. The method of claim 44, wherein said array of points comprises a back-lit, planar surface having a plurality of apertures arranged in an array and separated from each other by a known distance.

48. The method of claim 43, wherein said scanner comprises a hand-held scanner.

49. The method of claim 47, wherein said scanner remains fixed in position and wherein said flat planar surface and said back-lit surface are moved back and forth relative to said scanner to achieve said first and second distances.

50. A calibration apparatus for a scanner, said scanner having a projection system for projecting a pattern and a electronic imaging device for capturing images of said pattern after reflection of said pattern off an object, comprising:

at least one calibration surface comprising a Z calibration surface and an X-Y calibration surface, and wherein the calibration station further comprises;

a carrier system holding said Z calibration surface and said X-Y calibration surface and moving said Z calibration surface and said X-Y calibration surface relative to said scanner between two different distances from said scanner;

said carrier system further comprising a drive mechanism for moving either said Z calibration surface or said X-Y calibration surface into the optical path of said scanner.

51. The calibration station of claim 50, wherein said Z calibration surface comprises a smooth, flat, planar reflective surface.

52. The calibration station of claim 50, wherein said X-Y calibration surface comprises a back-lit, planar surface having a plurality of apertures arranged in an array and separated from each other by a known distance.

53. The calibration station of claim 50, wherein said X-Y calibration surface comprises planar surface having a plurality of light sources arranged in an array and separated from each other by a known distance.

54. A machine-readable memory for a scanner used to calculate three dimensional information of an object scanned by said scanner, said scanner having a projection system for projecting a pattern onto an object and a electronic imaging device for imaging said pattern after reflection of said pattern from said object, said memory comprising:

an array of data storage locations containing a three-axis (X, Y and Z) coordinate system calibration relationship for said scanner,

said calibration relationship identifying;

(1) pixel coordinates for said electronic imaging device for numerous portions of said pattern, said pixel coordinates associated with distance information from said projection system in a Z direction at at least two different Z distances, and (2) distance information in X and Y directions, for said numerous portions of said pattern, at said at least two different Z distances.

55. The machine-readable memory of claim 54, wherein said calibration relationship comprises a calibration table.

56. The machine-readable memory of claim 54, wherein said memory is incorporated into a scanning work station coupled to said scanner.

57. The machine-readable memory of claim 54, wherein said memory is incorporated into said scanner.

58. The machine-readable memory of claim 54, wherein said memory is incorporated into a computer remote from said scanner and receiving data from said scanner, said computer calculating surface geometry of an object scanned by said scanner in accordance with said table.

59. A machine-readable memory accessible by a computing device, said memory comprising data storage regions storing surface information in three dimensions of at least a portion of a work of art, said surface information obtained by scanning said work of art with a scanner and calculating said surface information in three dimensions from a series of images obtained by said scanner;

wherein said scanner comprises a projection system for projecting a pattern and an electronic imaging device for generating an image of said pattern after reflection of said pattern from an object, and wherein said scanner is calibrated using a method comprising the steps of;

projecting said pattern onto at least one surface at two different distances from said scanner and generating first and second images with said electronic imaging device of said pattern at said two distances;

imaging a set of objects of known spatial X-Y relationship at said two different differences;

generating a three-axis (X, Y and Z) coordinate system calibration relationship for said scanner, said calibration relationship indicating;

(1) pixel coordinates for numerous portions of said pattern when said pattern is projected onto said surface at said two different distances, and (2) distance information in X and Y directions for said portions of said pattern for said two different distances; and

storing said calibration relationship in a memory associated with said scanner.

60. The machine-readable memory of claim 59, wherein said memory comprises a library of surface information in three dimensions for a plurality of works of art.

61. The machine-readable memory of claim 59, wherein said scanner comprises a hand-held scanner.

63. The machine-readable memory of claim 59, wherein said work of art comprises a sculpture.

62. The machine-readable memory of clam 60, wherein said work of art is authenticated using said library of said plurality of works of art.

64. A calibration device for a scanner, wherein said scanner has a projection system for projecting a pattern and an electronic imaging device for capturing images of said pattern after reflection of said pattern off an object, the calibration device comprising:

a Z calibration surface receiving said projected pattern moveable relative to said scanner between at least two Z distances separated by a known amount; and

a plurality of detectable features of known X-Y spatial relationship positioned with respect to said scanner at a known Z distance relative to said Z calibration surface.

66. The calibration device of claim 64, wherein said Z calibration surface comprises a planar surface.

67. The calibration device of claim 64 or claim 65, wherein said Z calibration surface includes said plurality of detectable features of known X-Y spatial relationship.

68. The calibration device of claim 64 or claim 65, wherein said Z calibration surface and said plurality of detectable features of known X-Y spatial relationship comprise distinct objects.

69. The calibration device of claim 64 or claim 65, wherein said plurality of detectable feature comprise 4 or more detectable features.

70. The calibration device of claim 64 or claim 65 wherein said plurality of detectable features comprise an array of points placed on a surface.

71. The calibration device of claim 64 or claim 65, wherein said plurality of detectable features of known X-Y spatial relationship are printed on an object.

72. The calibration device of claim 64 or claim 65, wherein said plurality of detectable features of known X-Y spatial relationship are formed on a surface of an object by a laser.

73. The calibration device of claim 64 or claim 65, wherein said Z calibration surface comprises a curved surface.

75. The calibration device of claim 64 or claim 65, wherein said calibration device is used to calibrate a scanning system in which the spatial relationship of the projection and imaging axes is not known precisely.

76. The calibration device of claim 64 or claim 65, wherein said Z calibration surface has a spatial extent in X and Y directions such that the field of view of said electronic imaging device is completely filled by said Z calibration surface at said two different distances.

77. The calibration device of claim 64 or claim 65, wherein said two Z distances is such that the combined depth of focus of the imaging device and the projection device encompasses said two Z distances.

78. The calibration device of claim 64 or claim 65, wherein said plurality of detectable features of known X-Y spatial relationship are arranged on an object such that they are spatially extending in X and Y directions on said object at said two different distances at least equal to the field of view of said electronic imaging device.

79. The calibration device of claim 64 or claim 65, wherein said plurality of detectable features of known X-Y spatial relationship comprise an array of dots printed on a surface.

65. A calibration device for a scanner, wherein said scanner has a projection system for projecting a pattern and an electronic imaging device for capturing images of said pattern after reflection of said pattern off an object, the calibration device comprising:

a Z calibration surface receiving said projected pattern and having a spatial extent reflecting said pattern at at least two Z distances separated by a known amount; and

a plurality of detectable features of known X-Y spatial relationship positioned at at least two known Z distances relative to said Z calibration surface.

74. The calibration device of claim 65, wherein said Z calibration surface comprises an plurality of planar parallel surfaces.