Methods and systems for determining the pose of a camera with respect to at least one object of a real environment

• US 9,934,612 B2
• Filed: 02/27/2015
• Issued: 04/03/2018
• Est. Priority Date: 02/13/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Meier, Peter, Himane, Selim Ben, Misslinger, Stefan, Blachnitzky, Ben
• Primary Examiner(s): Rao, Andy
• Application Number:
  US14/633,386
• Publication Number:
  US 20150310666A1
• Time in Patent Office: 1,131 Days
• Field of Search: 348 36- 39, 348 42- 60, 348135-140
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/20092 Interactive image processing based on input by user
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2210/04 Architectural design, interior design
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: H04N 5/2224 related to virtual studio applications

Abstract

Method for determining the pose of a camera with respect to an object of a real environment for use in authoring/augmented reality application that includes generating a first image by the camera capturing a real object of a real environment, generating first orientation data from at least one orientation sensor associated with the camera or from an algorithm which analyzes the first image for finding and determining features which are indicative of an orientation of the camera, allocating a distance of the camera to the real object, generating distance data indicative of the allocated distance, determining the pose of the camera with respect to a coordinate system related to the real object of the real environment using the distance data and the first orientation data. May be performed with reduced processing requirements and/or higher processing speed, in mobile device such as mobile phones having display, camera and orientation sensor.

Claims

1. A method for superimposing a virtual model with a real environment, comprising:

receiving, from a camera, an image depicting a real environment;

generating first orientation data describing a vector normal to a 3D plane within a reference coordinate system;

placing a virtual model at a first location within the reference coordinate system, wherein the virtual model represents an object;

determining first position data indicative of at least one distance between the at least one camera and the 3D plane within the reference coordinate system;

moving the virtual model to a second location in the reference coordinate system, the second location positioned on the 3D plane defined by the first orientation data and the first position data;

modifying a dimension of the virtual model to generate a modified virtual model such that a first projection of the virtual model onto the image from the first location substantially equals a second projection of the modified virtual model onto the image from the second location; and

generating a modified version of an additional image of the real environment by projecting the modified virtual model from the second location onto the additional image of the real environment.

2. The method according to claim 1, wherein a camera pose associated with capturing the additional image is different from a camera pose associated with capturing the image.

3. The method according to claim 1, wherein modifying the dimension of the virtual model comprises scaling a size of the virtual model.

4. The method according to claim 1, wherein modifying the dimension of the virtual model comprises scaling or moving a coordinate system related to the virtual model.

5. The method according to claim 1, wherein the additional image is received from an additional camera that is distinct from the camera.

6. The method according to claim 1, wherein the additional image is received from the camera.

7. The method according to claim 1, wherein the first orientation data is generated based on features of the image or data received from an orientation sensor associated with the camera.

8. The method according to claim 1, wherein the 3D plane is parallel to an Earth ground plane.

9. The method according to claim 1, wherein the first position data is determined based on data received from a distance sensor associated with the camera.

10. The method according to claim 1, wherein

the reference coordinate system is associated with a position where the camera captures the image;

or

the reference coordinate system is associated with at least part of the real environment.

11. The method according to claim 1, wherein moving the virtual model to be positioned on the 3D plane further comprises:

identifying a line defined by a center of the camera and a point of the virtual model; and

moving the virtual model along the line.

12. The method according to claim 1, wherein the virtual model is placed at the first location within the reference coordinate system according to the first orientation data.

13. The method of claim 1, further comprising generating second orientation data associated with the additional image using an orientation sensor associated with the camera.

14. The method of claim 1, wherein the first position data is determined based on features of the image and features of the additional image.

15. The method according to claim 1, further comprising moving the virtual model along the 3D plane.

16. A method for superimposing a virtual model with a real environment, comprising:

receiving, from a camera, an image depicting a real environment;

generating first orientation data describing a vector normal to a 3D plane within a reference coordinate system;

placing a virtual model at a first location within the reference coordinate system, wherein the virtual model represents an object;

determining first position data indicative of at least one distance between the camera and the 3D plane within the reference coordinate system;

casting at least one ray from a position in the reference coordinate system where the camera captured the image to at least one point of the virtual model placed at the first location;

identifying at least one intersection point where the at least one ray intersects the 3D plane; and

generating a modified version of an additional image by projecting the virtual model onto the additional image such that the at least one point of the virtual model is superimposed with the corresponding at least one intersection point.

17. The method according to claim 16, wherein the additional image is received from an additional camera that is distinct from the camera.

18. The method according to claim 16, wherein the first position data is generated based on:

data received from a distance sensor associated with the camera;

or

features of the image and the additional image.

19. The method according to claim 16, wherein the first orientation data is generated based on data received from an orientation sensor associated with the camera.

20. A system for superimposing a virtual model with a real environment, comprising a processing device coupled with at least one camera, wherein the processing device is configured to:

receive, from the at least one camera, an image depicting a real environment;

generate first orientation data describing a vector normal to a 3D plane within a reference coordinate system;

place a virtual model at a first location within the reference coordinate system, wherein the virtual model represents an object;

determine first position data indicative of at least one distance between the at least one camera and the 3D plane within the reference coordinate system;

move the virtual model to a second location in the reference coordinate system, the second location positioned on the 3D plane defined by the first orientation data and the first position data;

modify a dimension of the virtual model to generate a modified virtual model such that a first projection of the virtual model onto the image from the first location substantially equals a second projection of the modified virtual model onto the image from the second location; and

generate a modified version of an additional image of the real environment by projecting the modified virtual model from the second location onto the additional image.