Position and orientation determination in 6-DOF

• US 9,443,308 B2
• Filed: 12/05/2012
• Issued: 09/13/2016
• Est. Priority Date: 12/06/2011
• Status: Active Grant

• Current Assignee: Hexagon Technology Center GmbH (Hexagon AB)
• Original Assignee: Hexagon Technology Center GmbH (Hexagon AB)
• Inventors: Pettersson, Bo, Siercks, Knut, Voit, Eugen, Hinderling, Jrg, Zebhauser, Benedikt, Schneider, Klaus
• Primary Examiner(s): Koziol, Stephen R
• Assistant Examiner(s): Lee, Jonathan S
• Application Number:
  US14/363,102
• Publication Number:
  US 20140286536A1
• Time in Patent Office: 1,378 Days
• US Class Current: 1/1
• CPC Class Codes: E02F 3/842 using electromagnetic, optical or photoelectric beams, e.g. laser beams
• CPC Class Codes: E02F 9/261 Surveying the work-site to be treated
• CPC Class Codes: G01C 15/002 Active optical surveying means optical plumbing G01C15/105
• CPC Class Codes: G01C 15/06 Surveyors' staffs; Movable markers
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

A method for a six degree of freedom position and orientation determination of a known shape in a scenery is disclosed. The method includes taking a range image with a range imaging camera and a visual picture with a digital camera. The range imaging camera includes a sensor array with a first number of pixels. Determining a 3D cluster of points from range information collected from the sensor array to a point of the scenery. the digital camera comprises an image sensor having a second number of pixels, resulting in a 2D picture. A stored 3D digital representation of the known shape may be fitted in a virtual space to match the reproduction of the known object in the 2D picture and the 3D cluster of points and determining the six degree of freedom position and orientation of the known shape according to the virtual match.

Claims

1. A method for a six degree of freedom position and orientation determination of a three dimensional known shape in a scenery, the method comprising:

taking a range image by means of a range imaging camera with a range image module having a sensor array with a first number of pixels, wherein for each of the first pixels a range information from the sensor to a point of the scenery is determined, resulting in a 3D cluster of points;

taking a visual picture with a digital camera with an image sensor having a second number of pixels, resulting in a 2D picture, wherein the 2D picture includes intensity and spectral information, wherein a relation of the first and the second pixels fields of view is known, wherein the second number of pixels is higher than the first number of pixels;

geometrically fitting of a stored 3D digital representation of the known shape in a virtual space to virtually match with the reproduction of the known shape in the taken 2D picture and the taken 3D cluster of points;

a first matching of the 3D digital representation with the visual picture and a second matching of the 3D digital representation with the range image and determining the six degree of freedom position and orientation by combining the results of the first and second matching; and

determining the six degree of freedom position and orientation of the known shape in the scenery according to the virtual match.

3. The method according to claim 1, further comprising extracting a 3D geometrical information by combining information from the visual picture and information from the range image, wherein the combining comprises a matching of at least one geometrical feature of the known shape in both of the range image and the visual picture and a further matching of the geometrical feature in the 3D digital representation within the combined information.

4. The method according to claim 1,

wherein the position and orientation by combining the results of the first and the second matching are determined according to an adjustment calculus or a maximum likelihood algorithm.

5. The method according to claim 1, wherein the matching comprises:

identifying a face by combining;

at least one of edge extraction of the face in the visual picture; and

face detection in the range image, preferably by fitting planes in the cluster of points, and

matching the identified face and edge with a corresponding face of the 3D digital representation according to a least square or maximum likelihood algorithm.

6. The method according to claim 1, further comprising recognition of textural structures within the visual picture on the known shape, comprising an additional position and orientation determination according to the appearing size of at least one of the known shape and/or its texture in the visual picture.

7. The method according to claim 1, further comprising taking the visual picture and the range image from a single measurement apparatus with a single line of sight, wherein the taking of the range image and the visual picture is synchronized in time.

8. The method according to claim 1, further comprising evaluating intensity information determined from the reflected intensity of the range imaging light received by the range camera as intensity values of the distance measurement light scattered back by the known shape, and including the intensity information in the virtual match.

9. The method according to claim 1, further comprising determining a dependent position and orientation of an item attached to the known shape, the known shape being a known three dimensional solid volumetric body having a three dimensional outer configuration facilitating a precisely and uniquely determinable spatial position and orientation of the known shape in six degrees of freedom.

10. The method according to claim 9, wherein the attached item is a tactile coordinate-measurement probe, and the method is measuring spatial coordinates of a measurement object.

11. The method according to claim 9, wherein the attached item is a robot arm which'"'"'s posture in space is determined according to the position and orientation of the known shape, and the method is determining position and orientation of the robots arm.

12. The method according to claim 9, wherein the attached item is a working tool of an earth moving machine which spatial coordinates are determined, wherein the earth moving machine is selected from the group consisting of a dozer, grader and digger, and the method includes determining position and orientation of the earth moving machine.

13. The method according to claim 1, further comprising using the known relation of the first and the second pixels in the step of fitting or determining.

14. A method of three dimensional modeling by digitalizing an outer form of a measurement object by:

a rough 3D modeling comprising;

a taking of a range image of the measurement object;

a taking of a visual picture of the measurement object; and

generating a rough 3D model by combining information of the range and the visual image;

a fine 3D modeling by;

measuring at least one surface point of the measurement object with a tactile probe, wherein the exact spatial position and orientation of the measurement point of the tactile probe is determined according to the method according to claim 1; and

refining measurement points of the rough model according to the at least one surface point, by a fine adjusting of at least one point of the rough 3D model according to the at least one surface point of the fine modeling.

15. A method of generating a three dimensional model of a measurement object, comprising the steps of:

determining the spatial position and orientation of a first known shape according to claim 1, wherein the first known shape is attached to a mirror and placed in a primary field of view of the range camera and the visual camera, the first known shape being a frame around the mirror and the mirror being reflective for the wavelengths of both of the visual camera and the range camera;

determining a secondary field of view of the range and visual camera, which is a subset of the primary field of view being redirected by the mirror, according to the position and orientation of the first known shape;

measuring in the secondary field of view by the range or visual camera, by position and orientation determination of a second known shape according to claim 1 in the secondary field of view;

transforming measurements taken in the secondary field of view into a coordinate system of the first field of view according to the determined position and orientation of the mirror; and

generating the three dimensional model according to measurements from the primary and secondary field of view, for measuring sections of the measurement object which are not visible or not unambiguously measurable in the primary field of view.

17. The method according to claim 4, wherein at least one of multiple faces and edges are three dimensionally matched.

18. The method according to claim 5, having the purpose of deriving uniqueness in the position and orientation determination according to the textural information.

19. The method according to claim 15, wherein the measurement apparatus and the measurement object are steady with respect to each other and the mirror is moveable.

20. The method according to claim 15, wherein the measurement apparatus and the measurement object are steady with respect to each other and the mirror is moveable, for modeling the measurement object from side, behind, or inside.

2. A method for a six degree of freedom position and orientation determination of a three dimensional known shape in a scenery, the method comprising:

taking a range image by means of a range imaging camera with a range image module having a sensor array with a first number of pixels, wherein for each of the first pixels a range information from the sensor to a point of the scenery is determined, resulting in a 3D cluster of points;

taking a visual picture with a digital camera with an image sensor having a second number of pixels, resulting in a 2D picture, wherein the 2D picture includes intensity and spectral information, wherein a relation of the first and second pixels fields of view is known, wherein the second number of pixels is higher than the first number of pixels;

directional dithering of an optical axis of the range camera relative to an optical axis of the digital camera in a sub-resolution of the first pixels, introduced by means of a reversely used optical image stabilization unit, and combining multiple, directional dithered low resolution range images for increasing the resolution of the cluster of points beyond a resolution of the first pixels;

geometrically fitting of a stored 3D digital representation of the known shape in a virtual space to virtually match with the reproduction of the known shape in the taken 2D picture and the resolution-increased 3D cluster of points; and

determining the six degree of freedom position and orientation of the known shape in the scenery according to the virtual match.

21. The method according to claim 2, further comprising extracting a 3D geometrical information by combining information from the visual picture and information from the range image, wherein the combining comprises a matching of at least one geometrical feature of the known shape in both of the range image and the visual picture and a further matching of the geometrical feature in the 3D digital representation within the combined information.

22. The method according to claim 2, wherein the position and orientation by combining the results of the first and the second matching are determined according to an adjustment calculus or a maximum likelihood algorithm.

23. The method according to claim 2, wherein the matching comprises:

identifying a face by combining;

at least one of edge extraction of the face in the visual picture; and

face detection in the range image, preferably by fitting planes in the cluster of points, and

matching the identified face and edge with a corresponding face of the 3D digital representation according to a least square or maximum likelihood algorithm.

24. The method according to claim 2, further comprising recognition of textural structures within the visual picture on the known shape, comprising an additional position and orientation determination according to the appearing size of at least one of the known shape or its texture in the visual picture.

25. The method according to claim 2, further comprising taking the visual picture and the range image from a single measurement apparatus with a single line of sight, wherein the taking of the range image and the visual picture is synchronized in time.

26. The method according to claim 2, further comprising evaluating intensity information determined from the reflected intensity of the range imaging light received by the range camera as intensity values of the distance measurement light scattered back by the known shape, and including the intensity information in the virtual match.

27. The method according to claim 2, further comprising determining a dependent position and orientation of an item attached to the known shape, the known shape being a known three dimensional solid volumetric body having a three dimensional outer configuration facilitating a precisely and uniquely determinable spatial position and orientation of the known shape in six degrees of freedom.

28. The method according to claim 27, wherein the attached item is a tactile coordinate-measurement probe, and the method is measuring spatial coordinates of a measurement object.

29. The method according to claim 27, wherein the attached item is a robot arm which'"'"'s posture in space is determined according to the position and orientation of the known shape, and the method is determining position and orientation of the robots arm.

30. The method according to claim 27, wherein the attached item is a working tool of an earth moving machine which spatial coordinates are determined, wherein the earth moving machine is selected from the group consisting of a dozer, grader and digger, and the method includes determining position and orientation of the earth moving machine.

31. The method according to claim 2, further comprising using the known relation of the first and the second pixels in the step of fitting or determining.

32. A method of three dimensional modeling by digitalizing an outer form of a measurement object by:

a rough 3D modeling comprising;

a taking of a range image of the measurement object;

a taking of a visual picture of the measurement object; and

generating a rough 3D model by combining information of the range and the visual image;

a fine 3D modeling by;

measuring at least one surface point of the measurement object with a tactile probe, wherein the exact spatial position and orientation of the measurement point of the tactile probe is determined according to the method according to claim 2; and

refining measurement points of the rough model according to the at least one surface point, by a fine adjusting of at least one point of the rough 3D model according to the at least one surface point of the fine modeling.

33. A method of generating a three dimensional model of a measurement object, comprising the steps of:

determining the spatial position and orientation of a first known shape according to claim 2, wherein the first known shape is attached to a mirror and placed in a primary field of view of the range camera and the visual camera, the first known shape being a frame around the mirror and the mirror being reflective for the wavelengths of both of the visual camera and the range camera;

determining a secondary field of view of the range and visual camera, which is a subset of the primary field of view being redirected by the mirror, according to the position and orientation of the first known shape;

measuring in the secondary field of view by the range or visual camera, by position and orientation determination of a second known shape according to claim 2 in the secondary field of view;

transforming measurements taken in the secondary field of view into a coordinate system of the first field of view according to the determined position and orientation of the mirror; and

generating the three dimensional model according to measurements from the primary and secondary field of view, for measuring sections of the measurement obj ect which are not visible or not unambiguously measurable in the primary field of view.

34. The method according to claim 22, wherein at least one of multiple faces and edges are three dimensionally matched.

35. The method according to claim 23, having the purpose of deriving uniqueness in the position and orientation determination according to the textural information.

36. The method according to claim 33, wherein the measurement apparatus and the measurement object are steady with respect to each other and the mirror is moveable.

37. The method according to claim 33, wherein the measurement apparatus and the measurement object are steady with respect to each other and the mirror is moveable, for modeling the measurement object from side, behind, or inside.

16. A computer program product with program code being stored on a non-transitory machine readable medium, the program code being configured to automatically execute and operate the method comprising:

taking a range image by means of a range imaging camera with a range image module having a sensor array with a first number of pixels, wherein for each of the first pixels a range information from the sensor to a point of the scenery is determined, resulting in a 3D cluster of points;

taking a visual picture with a digital camera with an image sensor having a second number of pixels, resulting in a 2D picture, wherein the 2D picture includes intensity and spectral information, wherein a relation of the first and the second pixels fields of view is known, wherein the second number of pixels is higher than the first number of pixels;

geometrically fitting of a stored 3D digital representation of the known shape in a virtual space to virtually match with the reproduction of the known shape in the taken 2D picture and the taken 3D cluster of points;

a first matching of the 3D digital representation with the visual picture and a second matching of the 3D digital representation with the range image and determining the six degree of freedom position and orientation by combining the results of the first and second matching; and

determining the six degree of freedom position and orientation of the known shape in the scenery according to the virtual match.