Method of and system for determining inaccuracy information in an augmented reality system

• US 7,768,534 B2
• Filed: 12/21/2006
• Issued: 08/03/2010
• Est. Priority Date: 12/23/2005
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Metaio GmbH (Apple Inc.)
• Inventors: Meier, Peter, Pentenrieder, Katharina
• Primary Examiner(s): GOOD JOHNSON, MOTILEWA
• Application Number:
  US11/614,155
• Publication Number:
  US 20070146391A1
• Time in Patent Office: 1,321 Days
• Field of Search: 345/633, 345/619
• US Class Current: 345/633
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration

Abstract

In a method of and a system for determining inaccuracy information (600) in positioning a data model (1) of virtual information in an augmented reality system, the data model (1) is projected to the image plane of a display device (4) for mixing with visual impressions of a real environment. The inaccuracy information is calculated from parameters (10-0 to 50-2, 40, 50, 60) in a mapping computing process by means of which the data model (1) is projected to the image plane of the display device (4). It is thus possible to indicate relative precise inaccuracy information in positioning the data model in an augmented reality system.

Claims

1. A method of determining inaccuracy information (600) for providing a measurement statement in positioning a data model (1) of virtual information in an augmented reality system, comprising:

taking an image of at least part of the real environment by means of a camera (2, 3);

projecting said data model (1) to an image plane of a display device (4) for mixing with the image of the at least part of the real environment;

using a tracking system (100) in obtaining tracking information (10-0, 10-1, 10-2) for defining a virtual coordinate system (20, 30) based on which the data model is projected to the image plane of the display device and serving to map said data model (1) at a specific position in said image;

processing data of said tracking system (100);

processing data of a camera calibration (200) providing information on camera parameters of said camera;

processing situation-specific data (300) providing information on representation of said data model (1) with respect to the virtual coordinate system (20, 30);

computing by a data processing device said inaccuracy information from an uncertainty in determining the virtual coordinate system, an uncertainty in determining the camera parameters and uncertainties arising in the relative positioning of the data model with respect to the virtual coordinate system.

2. The method of claim 1 further comprising:

wherein said display device is implemented as a see-through display device (4);

wherein said camera calibration comprises see-through calibration (200).

3. The method of claim 2 wherein

said data of said tracking system (100) contains information on measurement inaccuracy (10-1, 10-2) of said tracking system;

said data of said camera calibration (200) or said see-through calibration contain information on measurement inaccuracy of said camera calibration or said see-through calibration, respectively; and

,

said situation-specific data (300) contains information on inaccuracy of said representation of said data model with respect to said virtual coordinate system.

4. The method of claim 1 wherein said inaccuracy information (600) contains information on an uncertainty, particularly defining a variance.

5. The method of claim 1 wherein said inaccuracy information (600) contains an error value.

6. The method of claim 1 wherein said inaccuracy information (600) is determined via the following propagation steps:

determining (400-1) positioning inaccuracy information for at least a selected structure (40) of said data model (1) in three-dimensional space; and

,

projecting (400-2) said positioning inaccuracy information of said selected structure (40) of said data model in said image plane (4).

7. The method of claim 6 wherein said determining (400-1) said positioning inaccuracy information in said three-dimensional space comprises:

processing data (10-1, 10-2) of a tracking system (100) containing information on a measurement inaccuracy of said tracking system as well as situation-specific data (300) containing information on an inaccuracy of representation of said data model (1) with respect to a virtual coordinate system (20, 30).

8. The method of claim 6 wherein said projecting (400-2) of said positioning inaccuracy information of said selected structure (40) of said data model to said image plane (4) is carried out in consideration of intrinsic camera parameters or in consideration of parameters of a see-through display device.

9. The method of claim 6 wherein subsequent to said projection (400-2) of said positioning inaccuracy information to said image plane, propagating (400-3) is carried out for taking into consideration a distortion process in projecting said positioning inaccuracy information.

10. The method of claim 6 further comprising propagating (400-4) in which a conversion of said projected positioning inaccuracy information into a pixel representation is carried out.

11. The method of claim 6 further comprising:

processing tracking information (100) containing data on an uncertainty (10-2) of said tracking information;

processing calibration data (200) of a camera calibration or see-through calibration containing data on an uncertainty of camera parameters or parameters of a see-through display device, respectively; and

,

processing situation-specific data (300) containing data on an uncertainty (20-2, 20-3) of representation of said data model (1) with respect to a virtual coordinate system (20, 30).

12. The method of claim 6 wherein said positioning inaccuracy information (600) contains information on a measurement uncertainty, particularly defining a variance.

13. The method of claim 6 wherein said positioning inaccuracy information (600) contains a positioning error.

14. The method of claim 1 wherein processed inaccuracy information (500) is generated via re-projecting said inaccuracy information (600) to three-dimensional space.

15. The method of claim 1 wherein said inaccuracy information (600) contains a statement on an absolute position of a point (41, 51, 61), a contour or an area of said data model (1).

16. The method of claim 1 wherein said inaccuracy information (600) contains a statement on a relative position of a point (40, 50, 60), a contour or an area of said data model (1) with respect to another structure.

17. The method of claim 1 further comprising:

providing tracking information (100) for defining a virtual coordinate system (20, 30) upon which said data model (1) is projected to said image plane of said display device (4), said virtual coordinate system (20, 30) being determined via a transformation (10-0) with respect to a global reference coordinate system (10) of a tracking sensor;

providing calibration data (200) for providing intrinsic camera parameters of a camera or parameters of a see-through display device; and

,

providing situation-specific data (300) providing information on representation of said data model (1) with respect to said virtual coordinate system (20, 30) providing translation information (30-0) describing a selected structure (40) of said data model, for which inaccuracy information is to be generated, in said virtual coordinate system (20, 30).

18. The method of claim 17 wherein said tracking information (100) contains data on an error (10-1) of said transformation (10-0), and/or said calibration data (200) contain data on an error of said intrinsic camera parameters or said parameters of said see-through display device, respectively, and/or said situation-specific data (300) contains data on an error (20-1, 30-1) of said translation information (30-0).

19. The method of claim 17 wherein said tracking information (100) contains data on an uncertainty (10-2) of said transformation (10-0) or of a related error (10-1), said calibration data (200) contains data on an uncertainty of said intrinsic camera parameters or said parameters of said see-through display device, respectively, or of a related error, and said situation-specific data (300) contains data on an uncertainty (20-2, 30-2) of said translation information (30-0) or of a related error (30-1).

20. A method of determining inaccuracy information (600) for providing a measurement statement in positioning a data model (1) of virtual information in an augmented reality system comprising:

projecting said data model (1) to an image plane of a display device (4); and

, mixing with visual impressions of a real environment, with said inaccuracy information being computed from parameters (10-0 to 50-2, 40, 50, 60) in a mapping computing process from which said data model (1) is projected to said image plane of said display device (4) in which an image is taken of at least part of the real environment by means of a camera (2, 3) and said mapping computing process is carried out using a tracking system (100) to obtain tracking information (10-0, 10-1, 10-2) serving to map said data model (1) at a specific position in said image further comprising

processing data of said tracking system (100),

processing data of a camera calibration (200) providing information on camera parameters of said camera, and

processing situation-specific data (300) providing information on representation of said data model (1) with respect to a virtual coordinate system (20, 30).

21. A method of determining inaccuracy information (600) for providing a measurement statement in positioning a data model (1) of virtual information in an augmented reality system comprising:

projecting said data model (1) to an image plane of a display device (4); and

, mixing with visual impressions of a real environment, with said inaccuracy information being computed from parameters (10-0 to 50-2, 40, 50, 60) in a mapping computing process on the basis of which said data model (1) is projected to said image plane of said display device (4), in which, by means of a see-through display device (4), said data model (1) is mixed with at least part of the real environment and said mapping computing process is carried out using a tracking system (100) to obtain tracking information (10-0, 10-1, 10-2) serving to map said data model (1) at a specific position in said display device, further comprising processing data of said tracking system (100),

processing data of a see-through calibration (200) providing information on parameters of said see-through display device (4), and

processing situation-specific data (300) providing information on representation of said data model (1) with respect to a virtual coordinate system (20, 30).

22. A system for determining inaccuracy information for a measurement statement in positioning a data model (1) of virtual information in an augmented reality system, comprising

a camera (2, 3) configured to take an image of at least part of the real environment;

a display device (4) configured to display a projection of said data model (1) to an image plane of said display device (4) and to mix said projection of said data model (1) with the image of the at least part of the real environment;

a tracking system (100) configured to obtain tracking information (10-0, 10-1, 10-2) that defines a virtual coordinate system (20, 30) based on which the data model is projected to the image plane of the display device and further configured to map said data model (1) at a specific position in said image;

said tracking system further configured to process data of said tracking system;

said tracking system further configured to process data of a camera calibration (200) that provides information on camera parameters of said camera;

said tracking system further configured to process situation-specific data (300) that provides information on representation of said data model (1) with respect to the virtual coordinate system (20, 30);

a data processing device configured to compute said inaccuracy information from an uncertainty in determining the virtual coordinate system, an uncertainty in determining the camera parameters and uncertainties that arise in a relative position of the data model with respect to the virtual coordinate system.

23. The system of claim 22 wherein said wherein said display device is implemented as a see-through display device (4) and wherein said camera calibration comprises see-through calibration (200).