Coordinate measuring device with a six degree-of-freedom handheld probe and integrated camera for augmented reality

• US 9,402,070 B2
• Filed: 06/08/2015
• Issued: 07/26/2016
• Est. Priority Date: 06/12/2014
• Status: Active Grant

• Current Assignee: Faro Technologies Incorporated
• Original Assignee: Faro Technologies Incorporated
• Inventors: Tohme, Yazid, Bridges, Robert E.
• Primary Examiner(s): Luo, Kate
• Application Number:
  US14/733,130
• Publication Number:
  US 20150365653A1
• Time in Patent Office: 414 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G01B 11/002 for measuring two or more coordinates
• CPC Class Codes: G01B 11/14 for measuring distance or clearance between spaced objects or spaced apertures G01B11/26 takes precedence; rangefinders G01C3/00
• CPC Class Codes: G01S 17/66 Tracking systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 7/4813 Housing arrangements
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals

Abstract

A method of combining 2D images into a 3D image includes providing a coordinate measurement device and a six-DOF probe having an integral camera associated therewith, the six-DOF probe being separate from the coordinate measurement device. In a first instance, the coordinate measurement device determines the position and orientation of the six-DOF probe and the integral camera captures a first 2D image. In a second instance, the six-DOF probe is moved, the coordinate measurement device determines the position and orientation of the six-DOF probe, and the integral camera captures a second 2D image. A cardinal point common to the first and second image is found and is used, together with the first and second images and the positions and orientations of the six-DOF probe in the first and second instances, to create the 3D image.

Claims

1. A method of combining a plurality of two-dimensional (2D) images into a three-dimensional (3D) image, the method comprising steps of:

providing a six-degree of freedom (six-DOF) probe assembly, the six-DOF probe assembly including a probe stylus and a probe head, the probe stylus including a probe tip, the probe tip having a spherical shape over a portion of its surface, the spherical shape having a probe center, the probe head including a retroreflector and an integral camera;

providing a coordinate measurement device having a device frame of reference, the device being separate from the six-DOF probe assembly, the coordinate measurement device including an orientation sensor, a first motor, a second motor, a first angle measuring device, a second angle measuring device, a distance meter, a position detector, a control system, and a processor, the orientation sensor configured to measure three orientational degrees of freedom of the six-DOF probe assembly, the first motor and the second motor configured together to direct a first beam of light to a first direction, the first direction determined by a first angle of rotation about a first axis and a second angle of rotation about a second axis, the first angle of rotation produced by the first motor and the second angle of rotation produced by the second motor, the first angle measuring device configured to measure the first angle of rotation and the second angle measuring device configured to measure the second angle of rotation, the distance meter configured to measure a distance from the coordinate measurement device to the retroreflector based at least in part on a first part of the first beam of light reflected by the retroreflector and received by a first optical detector and on a speed of light in air, the position detector configured to receive a second part of the first beam of light reflected by the retroreflector and to produce a first signal in response, the control system configured to send a second signal to the first motor and a third signal to the second motor, the second signal and the third signal based at least in part on the first signal, the control system configured to adjust the first direction of the first beam of light to a position in space of the retroreflector, the processor configured to determine, in the device frame of reference, 3D coordinates of the probe center, 3D coordinates of the camera, and the three orientational degrees of freedom of the six-DOF probe assembly;

in a first instance;

with the device, measuring a third angle with the first angle measuring device, measuring a fourth angle with the second angle measuring device, measuring with the distance meter a first distance, and measuring the three orientational degrees of freedom to obtain a first set of three orientational degrees of freedom;

forming a first 2D image with the camera;

in a second instance;

moving the six-DOF probe assembly;

with the device, measuring a fifth angle with the first angle measuring device, measuring a sixth angle with the second angle measuring device, measuring with the distance meter a second distance, and measuring the three orientational degrees of freedom to obtain a second set of three orientational degrees of freedom;

forming a second 2D image with the camera;

determining a first cardinal point in common between the first and second 2D images, the first cardinal point having a first location on the first 2D image and a second location on the second 2D image;

determining 3D coordinates of the first cardinal point in a first frame of reference based at least in part on the first angle, the second angle, the third angle, the fourth angle, the first distance, the second distance, the first set of three orientational degrees of freedom, the second set of three orientational degrees of freedom, the first location, and the second location;

creating the 3D image as a first composite 3D image from the first 2D image and the second 2D image based at least in part on the first 2D image, the second 2D image, and the 3D coordinates of the first cardinal point in the first frame of reference; and

storing the first composite 3D image.

2. The method of claim 1 wherein:

the steps of the first instance further include touching the probe tip to a first surface contact point, the first surface contact point associated with a first imaged contact point on the first 2D image and a second imaged contact point on the second 2D image;

measuring with the device the first 3D coordinates of the first surface contact point;

the steps of the second instance further include touching the probe tip to a second surface contact point, the second surface contact point associated with a third imaged contact point on the first 2D image and a fourth imaged contact point on the second 2D image;

measuring with the device the second 3D coordinates of the second surface contact point; and

rescaling the dimension of length of the first composite 3D image based at least in part on the first imaged contact point, the second imaged contact point, the third imaged contact point, the fourth imaged contact point, the first 3D coordinates, and the second 3D coordinates.

3. The method of claim 1 further including a step of measuring a first plurality of points on a surface of the object.

4. The method of claim 3 further including a step of forming a first graphical representation, the first graphical representation including the first composite 3D image and the first plurality of points on the surface of the object, the first plurality of points viewed from a first observer perspective relative to the first composite 3D image.

5. The method of claim 4 further including a step of forming a second graphical representation, the second graphical representation including the first composite 3D image and the first plurality of points on the surface of the object, the first plurality of points viewed form a second observer perspective relative to the first composite 3D image, the second observer perspective being different than the first observer perspective.

6. The method of claim 1 wherein:

in the step of forming a first 2D image with the camera, the first 2D image is a color image;

in the step of forming a second 2D image with the camera, the second 2D image is a color image; and

in the step of creating a first composite 3D image, the first composite 3D image is a color image.

7. The method of claim 1 wherein, in the step of forming a first 2D image with the camera, the first 2D image includes a first area of the object surface.

8. The method of claim 7 further including:

a step of providing a drawing; and

a step of superimposing the first composite 3D image onto the drawing.

9. The method of claim 1 further including:

providing a first graphical element; and

superimposing the first graphical element on the first composite 3D image.

10. The method of claim 9 wherein, in the step of providing a first graphical element, the first graphical element is a mechanical extension of the object.

11. The method of claim 9 wherein, in the step of providing a first graphical element, the first graphical element is placed within a volume represented by the first composite 3D image.

12. The method of claim 9 wherein, in the step of providing a first graphical element, the first graphical element is downloaded from a network connection.

13. The method of claim 7 further includes:

a step of forming a third 2D image with the camera, the third 2D image including a second area of the object surface, the second area including a region of the object not visible in the first composite 3D image; and

a step of forming a second composite 3D image, the second composite 3D image including the third 2D image and at least a portion of the first 2D image or a portion of the second 2D image.