Methods and systems for determining the pose of a camera with respect to at least one object of a real environment

• US 8,970,690 B2
• Filed: 02/13/2009
• Issued: 03/03/2015
• Est. Priority Date: 02/13/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Metaio GmbH (Apple Inc.)
• Inventors: Meier, Peter, Ben Himane, Selim, Misslinger, Stefan, Blachnitzky, Ben
• Primary Examiner(s): Nguyen, Thuong
• Application Number:
  US12/371,490
• Publication Number:
  US 20100208057A1
• Time in Patent Office: 2,209 Days
• Field of Search: 348/135, 348/416.1, 348/208.14, 345/158, 345/633
• US Class Current: 348/135
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/20092 Interactive image processing based on input by user
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2210/04 Architectural design, interior design
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: H04N 5/2224 related to virtual studio applications

Abstract

Method for determining the pose of a camera with respect to at least one object of a real environment for use in authoring/augmented reality application that includes generating a first image by the camera capturing a real object of a real environment, generating first orientation data from at least one orientation sensor associated with the camera or from an algorithm which analysis the first image for finding and determining features which are indicative of an orientation of the camera, allocating a distance of the camera to the real object, generating distance data indicative of the allocated distance, determining the pose of the camera with respect to a coordinate system related to the real object of the real environment using the distance data and the first orientation data. May be performed with reduced processing requirements and/or higher processing speed, in mobile device such as mobile phones having display, camera and orientation sensor.

Claims

1. A method for determining the pose of a camera with respect to at least one object of a real environment for use in an authoring or augmented reality application, comprising:

generating a first image comprising at least one image by a camera capturing a real object of a real environment;

generating first orientation data describing a normal to a ground plane within a coordinate system from at least one orientation sensor associated with the camera or from an algorithm which analyses the first image to find and determine features which are indicative of an orientation of the camera;

obtaining a virtual reference model within said coordinate system, wherein said virtual reference model represents a known sized object;

displaying said virtual reference model superimposed with the real environment at a position in the first image and providing an initial distance from the camera to the virtual reference model;

receiving user instructions to manipulate the virtual reference model by a user, then modifying the position or orientation of the virtual reference model resulting in at least a change in the position of the virtual reference model within the real environment, including at least one of

moving the position of the virtual reference model within the first image on a plane, with the plane defined at least in part by the first orientation data,

zooming the virtual reference model to alter the position,

rotating the virtual reference model;

calculating a distance of the camera to the real object of the real environment displayed in the first image according to the position of the virtual reference model based on said manipulating of the virtual reference model by the user, and generating distance data that is indicative of the distance;

determining a pose of the camera with respect to said coordinate system associated with said virtual reference model and related to the real object of the real environment using the distance data and the first orientation data.

2. The method of claim 1, wherein the manipulating of the virtual reference model by the user may include at least one of:

touching the virtual reference model using a touchscreen, using two fingers and moving the two fingers away from each other to increase a size and moving the two fingers closer to each other to decrease the size of the virtual reference model;

touching the virtual reference model using a touchscreen with the two fingers and rotating the two fingers about one another in order to rotate the virtual reference model around an axis perpendicular to a ground plane;

touching the virtual reference model with at least one finger and moving the at least one finger in order to move the virtual reference model across the plane.

3. The method of claim 1, wherein manipulation of the virtual reference model by the user may include at least one of:

touching the virtual reference model using a touchscreen, using two fingers and moving the two fingers away from each other to move the virtual reference model closer to a viewer, wherein the move takes place on an assumed ground plane and parallel to an intersection of the assumed ground plane with a plane containing a camera center, x in a camera coordinate system being zero;

touching the virtual reference model using a touchscreen, using the two fingers and moving the two fingers closer to each other to move the virtual reference model farther from the viewer, wherein the move takes place on the assumed ground plane and parallel to the intersection of the assumed ground plane with the plane containing the camera center, x in the camera coordinate system being zero;

touching the virtual reference model using the touchscreen with the two fingers and rotating the two fingers about one another in order to rotate the virtual reference model around an axis perpendicular to the assumed ground plane;

touching the virtual reference model with at least one finger and moving the at least one finger up and down in order to move the virtual reference model parallel to a normal of the assumed ground plane;

touching the virtual reference model with the at least one finger and moving the at least one finger left and right in order to move the virtual reference model on the assumed ground plane and parallel to the intersection of the assumed ground plane with the plane containing the camera center, y in the camera coordinate system being zero.

4. The method of claim 1, wherein allocating the distance of the camera to the real object includes providing a measurement device associated with the camera that measures at least one parameter that is indicative of the distance between the camera and the real object, wherein the distance data is generated based on the at least one parameter.

5. The method of claim 4, wherein the providing the measurement device includes providing at least one of:

a focusing unit of the camera, an independent distance sensor, at least one time of flight camera, a stereo camera or cameras.

6. The method of claim 1, further comprising providing a parameter indicative of a rotation of the camera around an axis that is perpendicular to an Earth ground plane.

7. The method of claim 1, wherein the allocating the distance of the camera to the real object includes providing at least one parameter indicative of a distance between two features of the real environment which are visible in the first image.

8. The method of claim 1, further including displaying at least one virtual object superimposed with the real environment in the first image in accordance with the pose of the camera with respect to the coordinate system related to the real object of the real environment.

9. The method of claim 8, determining at least one placement coordinate system in the first image using the distance data and the first orientation data, wherein the at least one virtual object is superimposed with the real environment relative to the at least one placement coordinate system.

10. The method of claim 1, further comprising:

generating a second image by the camera capturing the real object of the real environment;

extracting at least one respective feature from the first image and the second image and matching the at least one respective feature therein providing at least one relation that is indicative of a correspondence between the first image and the second image; and

,

providing the pose of the camera with respect to the coordinate system related to the real object in the first image, and determining the pose of the camera with respect to a second coordinate system related to the real object in the second image using the pose of the camera with respect to the coordinate system related to the real object in the first image and the at least one relation.

11. The method of claim 10, further comprising generating second orientation data associated with the second image from the at least one orientation sensor.

12. The method of claim 10, further comprising:

determining at least one placement coordinate system in the first image and a first position and orientation thereof in the first image using the distance data and the first orientation data; and

,

allocating a second at least one placement coordinate system in the second image with a second position and orientation corresponding to the respective position and orientation in the first image using the at least one relation.

13. The method of claim 12, further comprising:

extracting at least one second respective feature from the first image and the second image and determining a ground plane in both the first image and the second image and moving a placement coordinate system to be positioned on the ground plane.

14. The method of claim 13, further comprising:

superimposing at least one virtual object with the real environment in accordance with the placement coordinate system in the first image;

superimposing the at least one virtual object with the real environment in accordance with the placement coordinate system positioned on the ground plane in the first image;

moving the placement coordinate system along the ground plane to be positioned in that a projection of the at least one virtual object in accordance with the placement coordinate system in the first image substantially equals or comes near the projection of the at least one virtual object in accordance with the placement coordinate system positioned on the ground plane.

15. The method of claim 14, further comprising superimposing the at least one virtual object with the real environment in accordance with the placement coordinate system on the ground plane.

16. The method of claim 10, wherein the providing the at least one relation further includes at least one of the following:

providing at least one parameter indicative of a movement of the camera between capturing the first image and the second image;

providing a second at least one parameter indicative of a distance between two features of the real environment which are visible in both the first image and the second image;

providing a third at least one parameter indicative of a distance between at least one feature of the real environment and the camera.

17. The method of claim 16, wherein providing the at least one parameter indicative of the movement of the camera between taking the first image and the second image includes providing a first location parameter of the camera when taking the first image and a second location parameter of the camera when taking the second image.

18. The method of claim 17, generating at least one of the first and second location parameters by at least one of a satellite locating system, and an elevation measuring device.

19. The method of claim 1, further comprising providing a virtual reference model which is displayed superimposed with the real environment in the first image, placing the virtual reference model on an assumed ground plane and identifying in the first image at least two features of the real environment on the assumed ground plane near the virtual reference model, wherein a distance between the at least two features is determined using a known dimension of the virtual reference model.

20. A method for determining the pose of a camera with respect to at least one object of a real environment for use in an authoring or augmented reality application, comprising:

generating at least one image by a camera capturing an object of a real environment;

generating orientation data within a coordinate system from at least one orientation sensor associated with the camera or from an algorithm which analyses the at least one image and finding and determining features which are indicative of an orientation of the camera;

providing a measurement device associated with the camera configured to measure at least one parameter that is indicative of distance between the camera and the object;

determining a pose of the camera with respect to said coordinate system related to the object of the real environment based on the at least one parameter and the orientation data;

obtaining a virtual reference model within said coordinate system, wherein said virtual reference model represents a known sized object;

displaying said virtual reference model within the image on a plane at a position, with the plane defined at least in part by the orientation data and the parameter that is indicative of distance; and

,

receiving user instructions to manipulate the virtual reference model by a user, then modifying the position or orientation of the virtual reference model resulting in at least a change in the position of the virtual reference model, including at least one of

moving the position of the virtual reference model within the at least one image on a plane, with the plane defined at least in part by the orientation data,

zooming the virtual reference model to alter the position,

rotating the virtual reference model.

21. A system for determining the pose of a camera with respect to at least one object of a real environment for use in an authoring or augmented reality system, comprising:

at least one camera configured to generate at least one image capturing at least one object of a real environment;

at least one orientation sensor associated with the at least one camera, configured to generate orientation data of the at least one camera within a coordinate system;

a measurement device associated with the at least one camera, configured to measure a distance between the at least one camera and the at least one object; and

,

a processing device coupled with the at least one camera, wherein the processing device is configured to

generate distance data from the distance, and

determine a pose of the at least one camera with respect to said coordinate system related to the at least one object of the real environment through use of the distance data and the orientation data;

obtain a virtual reference model within said coordinate system, wherein said virtual reference model represents a known sized object;

display said virtual reference model within the image on a plane at a position, with the plane defined at least in part by the orientation data and the distance data; and

,

receive user instructions to manipulate the virtual reference model by a user, to at least change the position of the virtual reference model, including at least one of an instruction to

move the position of the virtual reference model within the at least one image on a plane, with the plane defined at least in part by the orientation data,

zoom the virtual reference model to alter the position,

rotate the virtual reference model.

22. The system of claim 21, wherein the system is included in a mobile device.

23. The system of claim 22, wherein the mobile device is a mobile telephone.

24. A method for determining the pose of a camera with respect to at least one object of a real environment for use in an authoring or augmented reality application, comprising:

generating a first image comprising at least one image by a camera that captures a real object of a real environment;

generating first orientation data describing a normal to a ground plane within a coordinate system from at least one orientation sensor associated with the camera or from an algorithm which analyses the first image to find and determine features which are indicative of an orientation of the camera or by user-interaction;

obtaining a virtual reference model within said coordinate system, wherein said virtual reference model represents a known sized object;

displaying said virtual reference model superimposed with the real environment at a position in the first image and providing an initial distance from the camera to the virtual reference model;

receiving user instructions to manipulate the virtual reference model by a user, then modifying the position or orientation of the virtual reference model resulting in at least a change in the position of the virtual reference model within the real environment, including at least one of

moving the position of the virtual reference model within the at least one image on a plane, with the plane defined at least in part by the orientation data to alter the position,

zooming the virtual reference model to alter the position,

rotating the virtual reference model;

calculating a distance of the camera to the real object of the real environment displayed in the first image according to the position of the virtual reference model based on said manipulating of the virtual reference model by the user, and generating distance data that is indicative of an allocated distance of the camera to the real object;

determining a pose of the camera with respect to said coordinate system associated with said virtual reference model and related to the real object of the real environment in the first image using the distance data and the first orientation data;

generating a second image by the camera that captures the real object of the real environment;

extracting at least one respective feature from the first image and the second image and matching the at least one respective feature to provide at least one relation that is indicative of a correspondence between the first image and the second image;

providing the pose of the camera with respect to the coordinate system related to the real object in the first image, and determining the pose of the camera with respect to a second coordinate system related to the real object in the second image using the pose of the camera with respect to the coordinate system related to the real object in the first image and the at least one relation; and

,

extracting at least one second respective feature from the first image and the second image and determining a ground plane in both the first image and the second image and moving a placement coordinate system to be positioned on the ground plane.

25. The method of claim 24, further including

moving the placement coordinate system and indicating the pose of a virtual object in respect to a real object on the ground plane in that the projection of the virtual object in accordance with the original placement coordinate system in the first image substantially equals or comes near the projection of the virtual object in accordance with the placement coordinate system moved and positioned on the ground plane.

26. The method of claim 25, wherein allocating a distance of the camera to the real object of the real environment displayed in the first image includes scaling the virtual object by user interaction.