Method and system for ascertaining the position and orientation of a camera relative to a real object

• US 9,008,371 B2
• Filed: 07/03/2008
• Issued: 04/14/2015
• Est. Priority Date: 07/18/2007
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Metaio GmbH (Apple Inc.)
• Inventors: Meier, Peter
• Primary Examiner(s): Akhavannik, Hadi
• Application Number:
  US12/669,411
• Publication Number:
  US 20100239121A1
• Time in Patent Office: 2,476 Days
• Field of Search: 382/100, 382/106, 382/107, 382152-154, 345/419, 345/427, 345/619, 345/629, 345/633, 345/679, 356/139.03, 356/138, 356/12, 332/86, 705/757
• US Class Current: 382/106
• CPC Class Codes: G06T 2207/20101 Interactive definition of point of interest, landmark or seed
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/75 involving models

Abstract

The invention relates to a method for ascertaining the position and orientation of a camera (11) relative to a real object (12) for use in merging a virtual data model (13) with an image generated by the camera (11), said image comprising at least one part of the real object (12). The method comprises the following steps: disposing the camera (11) on a movable part (22) of a sensor system (20) which is coupleable to at least one probe (23) which is suitable to come in contact with at least one part of the real object (12), positioning the movable part (22) of the sensor system, with the camera (11) applied, in such a manner that the image is generatable by the camera, and generating position data (31) of the sensor system (20) with respect to the real object (12), and determining the position and orientation of the camera (11) on the basis of the position data (31, 32) of the sensor system (20).

Claims

1. A method for merging a virtual data model with an image generated by a camera, said image comprising at least one part of a real object, having the following steps:

disposing the camera on a movable part of a sensor system which is coupled at least temporarily to at least one probe which is suitable to come in contact with at least one part of the real object in a calibrating process;

positioning the movable part of the sensor system, with the camera applied, in such a manner that the image is generated by the camera, and generating position data of the sensor system with respect to the real object, wherein the movable part of the sensor system is a measuring arm; and

determining the position and orientation of the camera relative to the real object on the basis of the position data of the sensor system and merging a virtual data model representing a design drawing with the image generated by the camera, on the basis of the position and orientation of the camera;

whereby the virtual data model is superimposed largely precisely with the real object or relative deviations between the virtual data model and the real object are ascertained; and

wherein for determination of the position and orientation of the camera, a determination of the position of the camera with respect to the at least one probe is effected by hand-eye calibration.

2. The method according to claim 1, having the steps of:

disposing a fixed part of the sensor system in a fixed relation to the real object and setting up a sensor coordinate system as the frame of reference of the sensor system; and

determining a position of the sensor coordinate system with respect to the real object.

3. The method according to claim 2, wherein

the determination of the position of the sensor coordinate system with respect to the real object is effected by the probe being brought in contact with at least one part of the real object in at least one calibrating process.

4. The method according to claim 3, wherein

the at least one calibrating process is performed according to one of a plurality of stored calibrating algorithms.

5. The method according to claim 1, wherein

for determination of the position and orientation of the camera a determination of the position of the camera with respect to the probe is effected by hand-eye calibration.

7. The method according to claim 1, wherein

a quality of the match of the virtual data model to the real object is determined, in particular a measurement of at least one distance between at least one part of the real object and at least one part of the virtual data model is effected.

8. The method according to claim 7, wherein

a measurement of at least one distance between at least one part of the real object and at least one part of the virtual data model is effected by the user selecting a part of the virtual model in the image or of the real object by means of the probe so as to determine a first point, and subsequently the user selecting a corresponding part of the real object by means of the probe or of the virtual model in the image so as to determine a second point, whereby the distance to be determined is the distance between the first and second points.

9. The method according to claim 7, wherein

the determination of the quality of the match of the virtual data model to the real object is carried out using an edge-based or area-based tracking method.

10. The method according to claim 7, wherein

the determination of the quality of the match of the virtual data model to the real object comprises the step that the user or an implemented algorithm performs a comparison upon the merge of the real object and the virtual data model until at least one part of the data model superimposes a corresponding part of the real object largely correctly, whereby the measure of the performed comparison is employed as the basis for a statement about the quality of the merge.

11. The method according to claim 7, wherein

inaccuracy information is generated upon the positioning of the data model relative to the real object.

15. A non-transitory computer program product having software code portions which are suitable to perform a method according to claim 1 for determining the position and orientation of the camera on the basis of the position data of the sensor system, when they are stored in a system for ascertaining the position and orientation of the camera relative to the real object.

6. A method for merging a virtual data model with an image generated by the camera or display apparatus, said image comprising at least one part of a real object, having the following steps:

disposing a tracking system on a first movable part of a sensor system which is coupled at least temporarily to at least one probe which is suitable to come in contact with at least one part of the real object in a calibrating process, wherein the first movable part of the sensor system is a measuring arm;

disposing the camera or the display apparatus in fixed relation to a second movable part of the sensor system;

positioning the first movable part of the sensor system such that the second movable part of the sensor system is detectable by the tracking system,

positioning the second movable part of the sensor system with the camera or display apparatus in such a manner that the image is generated by the camera or the display apparatus, and generating first position data of the sensor system with respect to the real object and second position data of the tracking system with respect to the second movable part of the sensor system; and

determining the position and orientation of the camera or the display apparatus relative to the real object on the basis of the first and second position data and merging a virtual data model representing a design drawing with the generated image on the basis of the position and orientation of the camera or the display apparatus;

whereby the virtual data model is superimposed largely precisely with the real object or relative deviations between the virtual data model and the real object are ascertained; and

wherein for determination of the position and orientation of the camera or the display apparatus, a determination of the transformation between the tracking system and the at least one probe is effected by hand-eye calibration.

12. A system for merging a virtual data model with an image generated by a camera, comprising;

a sensor system having at least one movable part to which at least one probe is coupled at least temporarily which is suitable to come in contact with at least one part of the real object in a calibrating process, wherein the movable part is a measuring arm;

a camera which is disposed on the movable part of the sensor system and which is suitable to generate an image which comprises at least one part of the real object;

having an interface device in order to output position data of the sensor system with respect to the real object, for determining the position and orientation of the camera relative to the real object; and

a processing device coupled to the interface device and adapted for determining the position and orientation of the camera relative to the real object on the basis of the position data of the sensor system, and for merging a virtual data model representing a design drawing with the image generated by the camera, on the basis of the position and orientation of the camera, whereby the virtual data model is superimposed largely precisely with the real object or relative deviations between the virtual data model and the real object are ascertained, and wherein for determination of the position and orientation of the camera, a determination of the position of the camera with respect to the at least one probe is effected by hand-eye calibration.

13. The system according to claim 12, wherein

the camera, the processing device and a display device for displaying the image of the camera are executed as an integrated device which is attached to an adapter device which produces a mechanical connection between the probe and integrated device.

14. A system for merging a virtual data model with an image generated by a camera or a display apparatus, comprising:

a sensor system having at least one first movable part to which at least one probe is coupled at least temporarily which is suitable to come in contact with at least one part of the real object in a calibrating process, and having a second movable part, wherein the first movable part is a measuring arm;

a tracking system which is disposed on the first movable part of the sensor system;

a camera or a display apparatus which is disposed on the second movable part of the sensor system and which is suitable to generate an image which comprises at least one part of the real object;

whereby the tracking system is adapted to be disposed such that the second movable part of the sensor system is detectable by the tracking system;

having an interface device in order to output first position data (of the sensor system with respect to the real object and second position data of the tracking system with respect to the second movable part of the sensor system, for determining the position and orientation of the camera or the display apparatus relative to the real object; and

processing device coupled to the interface device adapted for determining the position and orientation of the camera or the display apparatus relative to the real object, and for merging a virtual data model representing a design drawing with the generated image on the basis of the position and orientation of the camera or the display apparatus, whereby the virtual data model is superimposed largely precisely with the real object or relative deviations between the virtual data model and the real object are ascertained, and wherein for determination of the position and orientation of the camera or the display apparatus, a determination of the transformation between the tracking system and the at least one probe is effected by hand-eye calibration.