METHODS AND SYSTEMS FOR DETERMINING THE POSE OF A CAMERA WITH RESPECT TO AT LEAST ONE OBJECT OF A REAL ENVIRONMENT

• US 20100208057A1
• Filed: 02/13/2009
• Published: 08/19/2010
• Est. Priority Date: 02/13/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Metaio GmbH (Apple Inc.)
• Inventors: Meier, Peter, Blachntizky, Ben, Misslinger, Stefan, Himane, Selim Ben
• Granted Patent: US 8,970,690 B2
• Time in Patent Office: Days
• US Class Current: 348/135
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/20092 Interactive image processing based on input by user
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2210/04 Architectural design, interior design
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: H04N 5/2224 related to virtual studio applications

Abstract

Method for determining the pose of a camera with respect to at least one object of a real environment for use in authoring/augmented reality application that includes generating a first image by the camera capturing a real object of a real environment, generating first orientation data from at least one orientation sensor associated with the camera or from an algorithm which analyses the first image for finding and determining features which are indicative of an orientation of the camera, allocating a distance of the camera to the real object, generating distance data indicative of the allocated distance, determining the pose of the camera with respect to a coordinate system related to the real object of the real environment using the distance data and the first orientation data. May be performed with reduced processing requirements and/or higher processing speed, in mobile device such as mobile phones having display, camera and orientation sensor.

Claims

1. A method for determining the pose of a camera with respect to at least one object of a real environment for use in an authoring or augmented reality application, comprising:

generating a first image comprising at least one image by a camera capturing a real object of a real environment;

generating first orientation data from at least one orientation sensor associated with the camera or from an algorithm which analyses the first image to find and determine features which are indicative of an orientation of the camera;

allocating a distance of the camera to the real object of the real environment displayed in the first image, and generating distance data that is indicative of the distance; and

,

determining a pose of the camera with respect to a coordinate system related to the real object of the real environment using the distance data and the first orientation data.

2. The method of claim 1, wherein the allocating the distance of the camera to the real object includes providing a virtual reference model which is displayed superimposed with the real environment in the first image and generating the distance data from the virtual reference model.

3. The method of claim 2, wherein the method further includes receiving user instructions for manipulating the virtual reference model by a user, the manipulating of the virtual reference model including at least one of moving the virtual reference model at a particular position within the first image on a plane, with the plane defined at least in part by the first orientation data, and changing a dimension or variable of the virtual reference model.

4. The method of claim 3, wherein the manipulating of the virtual reference model by the user may include at least one of:

touching the virtual reference model using a touchscreen, using two fingers and moving the two fingers away from each other to increase a size and moving the two fingers closer to each other to decrease the size of the virtual reference model;

touching the virtual reference model using a touchscreen with the two fingers and rotating the two fingers about one another in order to rotate the virtual reference model around an axis perpendicular to a ground plane;

touching the virtual reference model with at least one finger and moving the at least one finger in order to move the virtual reference model across the plane.

5. The method of claim 3, wherein manipulation of the virtual reference model by the user may include at least one of:

touching the virtual reference model using a touchscreen, using two fingers and moving the two fingers away from each other to move the virtual reference model closer to a viewer, wherein the move takes place on an assumed ground plane and parallel to an intersection of the assumed ground plane with a plane containing a camera center, x in a camera coordinate system being zero;

touching the virtual reference model using a touchscreen, using the two fingers and moving the two fingers closer to each other to move the virtual reference model farther from the viewer, wherein the move takes place on the assumed ground plane and parallel to the intersection of the assumed ground plane with the plane containing the camera center, x in the camera coordinate system being zero;

touching the virtual reference model using the touchscreen with the two fingers and rotating the two fingers about one another in order to rotate the virtual reference model around an axis perpendicular to the assumed ground plane;

touching the virtual reference model with at least one finger and moving the at least one finger up and down in order to move the virtual reference model parallel to a normal of the assumed ground plane;

touching the virtual reference model with the at least one finger and moving the at least one finger left and right in order to move the virtual reference model on the assumed ground plane and parallel to the intersection of the assumed ground plane with the plane containing the camera center, y in the camera coordinate system being zero.

6. The method of claim 1, wherein allocating the distance of the camera to the real object includes providing a measurement device associated with the camera that measures at least one parameter that is indicative of the distance between the camera and the real object, wherein the distance data is generated based on the at least one parameter.

7. The method of claim 6, wherein the providing the measurement device includes providing at least one of:

a focussing unit of the camera, an independent distance sensor, at least one time of flight camera, a stereo camera or cameras.

8. The method of claim 1, further comprising providing a parameter indicative of a rotation of the camera around an axis that is perpendicular to an Earth ground plane.

9. The method of claim 1, wherein the allocating the distance of the camera to the real object includes providing at least one parameter indicative of a distance between two features of the real environment which are visible in the first image.

10. The method of claim 1, further including displaying at least one virtual object superimposed with the real environment in the first image in accordance with the pose of the camera with respect to the coordinate system related to the real object of the real environment.

11. The method of claim 10, determining at least one placement coordinate system in the first image using the distance data and the first orientation data, wherein the at least one virtual object is superimposed with the real environment relative to the at least one placement coordinate system.

12. The method of claim 1, further comprising:

generating a second image by the camera capturing the real object of the real environment;

extracting at least one respective feature from the first image and the second image and matching the at least one respective feature therein providing at least one relation that is indicative of a correspondence between the first image and the second image; and

,

providing the pose of the camera with respect to the coordinate system related to the real object in the first image, and determining the pose of the camera with respect to a second coordinate system related to the real object in the second image using the pose of the camera with respect to the coordinate system related to the real object in the first image and the at least one relation.

13. The method of claim 12, further comprising generating second orientation data associated with the second image from the at least one orientation sensor.

14. The method of claim 12, further comprising:

determining at least one placement coordinate system in the first image and a position and orientation thereof in the first image using the distance data and the first orientation data; and

,

allocating a second at least one placement coordinate system in the second image with a second position and orientation corresponding to the respective position and orientation in the first image using the at least one relation.

15. The method of claim 12, wherein the providing the at least one relation further includes at least one of the following:

providing at least one parameter indicative of a movement of the camera between capturing the first image and the second image;

providing a second at least one parameter indicative of a distance between two features of the real environment which are visible in both the first image and the second image;

providing a third at least one parameter indicative of a distance between at least one feature of the real environment and the camera.

16. The method of claim 15, wherein providing the at least one parameter indicative of the movement of the camera between taking the first image and the second image includes providing a first location parameter of the camera when taking the first image and a second location parameter of the camera when taking the second image.

17. The method of claim 16, generating at least one of the first and second location parameters by at least one of a satellite locating system, and an elevation measuring device.

18. The method of claim 1, further comprising providing a virtual reference model which is displayed superimposed with the real environment in the first image, placing the virtual reference model on an assumed ground plane and identifying in the first image at least two features of the real environment on the assumed ground plane near the virtual reference model, wherein a distance between the at least two features is determined using a known dimension of the virtual reference model.

19. The method of claim 14, further comprising:

extracting at least one second respective feature from the first image and the second image for determining a ground plane in both the first image and the second image and moving a placement coordinate system to be positioned on the ground plane.

20. The method of claim 19, further comprising:

superimposing at least one virtual object with the real environment in accordance with the placement coordinate system in the first image;

superimposing the at least one virtual object with the real environment in accordance with the placement coordinate system positioned on the ground plane in the first image;

moving the placement coordinate system along the ground plane to be positioned in that a projection of the at least one virtual object in accordance with the placement coordinate system in the first image substantially equals or comes near the projection of the at least one virtual object in accordance with the placement coordinate system positioned on the ground plane.

21. The method of claim 20, further comprising superimposing the at least one virtual object with the real environment in accordance with the placement coordinate system on the ground plane.

22. A method for determining the pose of a camera with respect to at least one object of a real environment for use in an authoring or augmented reality application, comprising:

generating at least one image by a camera capturing a object of a real environment;

displaying the at least one image on an image display;

generating orientation data from at least one orientation sensor associated with the camera or from an algorithm which analyses the at least one image for finding and features which are indicative of an orientation of the camera;

providing a virtual reference model which is displayed superimposed with the real environment in the at least one image and generating distance data from the virtual reference model, wherein the distance data is indicative of an allocated distance of the camera to the object; and

,

determining a pose of the camera with respect to a coordinate system related to the object of the real environment using the distance data and the orientation data.

23. A method for determining the pose of a camera with respect to at least one object of a real environment for use in an authoring or augmented reality application, comprising:

generating at least one image by a camera capturing an object of a real environment;

generating orientation data from at least one orientation sensor associated with the camera or from an algorithm which analyses the at least one image for finding and determining features which are indicative of an orientation of the camera;

providing a measurement device associated with the camera for measuring at least one parameter that is indicative of distance between the camera and the object;

determining a pose of the camera with respect to a coordinate system related to the object of the real environment based on the at least one parameter and the orientation data.

24. A system for determining the pose of a camera with respect to at least one object of a real environment for use in an authoring or augmented reality system, comprising:

at least one camera configured to generate at least one image that captures at least one object of a real environment;

an image displaying device configured to display the at least one image;

at least one orientation sensor associated with the at least one camera, configured to generate orientation data of the at least one camera;

a processing device coupled with the at least one camera and with the image displaying device, the processing device and configured to

provide a virtual reference model which is displayed superimposed with the real environment in the at least one image and generate distance data from the virtual reference model, the distance data being indicative of an allocated distance of the at least one camera to the at least one object;

receive user instructions via a user interface for manipulation of the virtual reference model by user placement of the virtual reference model at a particular position within the at least one image;

determine a pose of the at least one camera with respect to a coordinate system related to the at least one object of the real environment through use of the distance data and the orientation data.

25. The system of claim 24, wherein the system is included in a mobile device.

26. The system of claim 25, wherein the mobile device is a mobile telephone.

27. A system for determining the pose of a camera with respect to at least one object of a real environment for use in an authoring or augmented reality system, comprising:

at least one camera configured to generate at least one image capturing at least one object of a real environment;

at least one orientation sensor associated with the at least one camera, configured to generate orientation data of the at least one camera;

a measurement device associated with the at least one camera, configured to measure a distance between the at least one camera and the at least one object; and

,

a processing device coupled with the at least one camera, wherein the processing device is configured to

generate distance data from the distance, and

determine a pose of the at least one camera with respect to a coordinate system related to the at least one object of the real environment through use of the distance data and the orientation data.

28. The system of claim 27, wherein the system is included in a mobile device.

29. The system of claim 28, wherein the mobile device is a mobile telephone.

30. A method for determining the pose of a camera with respect to at least one object of a real environment for use in an authoring or augmented reality application, comprising:

generating a first image comprising at least one image by a camera that captures a real object of a real environment;

generating first orientation data from at least one orientation sensor associated with the camera or from an algorithm which analyses the first image to find and determine features which are indicative of an orientation of the camera or by user-interaction;

allocating a distance of the camera to the real object of the real environment displayed in the first image, and generating distance data that is indicative of an allocated distance of the camera to the real object;

determining a pose of the camera with respect to a coordinate system related to the real object of the real environment in the first image using the distance data and the first orientation data;

generating a second image by the camera that captures the real object of the real environment;

extracting at least one respective feature from the first image and the second image and matching the at least one respective feature to provide at least one relation that is indicative of a correspondence between the first image and the second image;

providing the pose of the camera with respect to the coordinate system related to the real object in the first image, and determining the pose of the camera with respect to a second coordinate system related to the real object in the second image using the pose of the camera with respect to the coordinate system related to the real object in the first image and the at least one relation; and

,

extracting at least one second respective feature from the first image and the second image for determining a ground plane in both the first image and the second image and moving a placement coordinate system to be positioned on the ground plane.