Method for representing virtual information in a view of a real environment

• US 10,074,215 B2
• Filed: 02/06/2015
• Issued: 09/11/2018
• Est. Priority Date: 10/12/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Meier, Peter, Kuhn, Michael, Angermann, Frank
• Primary Examiner(s): Cobb, Michael J
• Application Number:
  US14/616,213
• Publication Number:
  US 20150154803A1
• Time in Patent Office: 1,313 Days
• Field of Search: None
• CPC Class Codes: G01S 19/01 Satellite radio beacon positioning systems transmitting time-stamped messages, e.g. GPS [Global Positioning System], GLONASS [Global Orbiting Navigation Satellite System] or GALILEO
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

A method for representing virtual information in a view of a real environment comprises providing a virtual object having a global position and orientation with respect to a geographic global coordinate system, with first pose data on the global position and orientation of the virtual object, in a database of a server, taking an image of a real environment by a mobile device and providing second pose data as to at which position and with which orientation with respect to the geographic global coordinate system the image was taken. The method further includes displaying the image on a display of the mobile device, accessing the virtual object in the database and positioning the virtual object in the image on the basis of the first and second pose data, manipulating the virtual object or adding a further virtual object, and providing the manipulated virtual object with modified first pose data or the further virtual object with third pose data in the database.

Claims

1. A method for representing image information in a view of a real environment on a mobile device, comprising:

capturing an image of the real environment;

determining an image pose of the image in a reference coordinate system;

receiving, from a server device, a reference image and a reference pose of the reference image from a reference database;

matching features of a real object depicted in the image to corresponding features of a real object depicted in the reference image from the reference database;

generating an updated image pose based at least in part on the reference pose in response to determining that the features of the real object depicted in the image match the corresponding features of the real object depicted in the reference image;

receiving, from the server device, an indication of an object pose of a virtual object, the object pose based on the reference coordinate system;

determining an overlay position in the image based on the updated image pose and the object pose;

displaying the virtual object overlaid at the determined overlay position in the image on a display device;

receiving input requesting manipulation of the virtual object within the image; and

sending, to the server device, a request to replace the object pose with an updated object pose based on the updated image pose and the input requesting manipulation of the virtual object.

2. The method of claim 1, wherein the input requesting manipulation of the virtual object corresponds to user input received via a user interface.

3. The method of claim 1, wherein the reference coordinate system is a geographic global coordinate system or a coordinate system associated with the real object depicted in the image.

4. The method of claim 1, further comprising:

capturing a second image of the real environment, the second image associated with a second image pose; and

overlaying the virtual object on the second image based on the second image pose and the updated object pose.

5. The method of claim 1, wherein the image is captured by a camera of a mobile device, the method further comprising determining the image pose based on global positioning system (GPS) data generated by a GPS sensor of the mobile device.

6. The method of claim 1, wherein the image is captured by a camera of a mobile device, the method further comprising determining the image pose based on an identifier detected by a wireless local area network (WLAN) adapter of the mobile device.

7. The method of claim 1, wherein the image is captured by a camera of a mobile device, the method further comprising determining the image pose based on an identifier detected by a radio frequency identifier (RFID) sensor of the mobile device.

8. A computer readable memory device comprising computer readable code executable by one or more processors to:

capture an image of a real environment;

determine an image pose of the image in a reference coordinate system;

receive, from a server device, a reference image and a reference pose of the reference image from a reference database;

match features of a real object depicted in the image to corresponding features of a real object depicted in the reference image from the reference database;

generate an updated image pose based at least in part on the reference pose in response to determining that the features of the real object depicted in the image match the corresponding features of the real object depicted in the reference image;

receive, from the server device, an indication of an object pose of a virtual object, the object pose based on the reference coordinate system;

determine an overlay position in the image based on the updated image pose and the object pose;

display the virtual object overlaid at the determined overlay position in the image on a display device;

receive input requesting manipulation of the virtual object within the image; and

send, to the server device, a request to replace the object pose with an updated object pose based on the updated image pose and the input requesting manipulation of the virtual object.

9. The computer readable memory device of claim 8, wherein the input requesting manipulation of the virtual object corresponds to user input received via a user interface.

10. The computer readable memory device of claim 8, wherein the reference coordinate system is a geographic global coordinate system or a coordinate system associated with the real object depicted in the image.

11. The computer readable memory device of claim 8, wherein the computer readable code is further executable by the one or more processors to:

capture a second image of the real environment, the second image associated with a second image pose; and

overlay the virtual object on the second image based on the second image pose and the updated object pose.

12. A system comprising:

one or more processors; and

a memory coupled to the one or more processors and comprising computer readable code executable by the one or more processors to;

capture an image of a real environment;

determine an image pose of the image in a reference coordinate system;

receive, from a server device, a reference image and a reference pose of the reference image from a reference database;

match features of a real object depicted in the image to corresponding features of a real object depicted in the reference image from the reference database;

generate an updated image pose based at least in part on the reference pose in response to determining that the features of the real object depicted in the image match the corresponding features of the real object depicted in the reference image;

receive, from the server device, an indication of an object pose of a virtual object, the object pose based on the reference coordinate system;

determine an overlay position in the image based on the updated image pose and the object pose;

display the virtual object overlaid at the determined overlay position in the image on a display device;

receive input requesting manipulation of the virtual object within the image; and

send, to the server device, a request to replace the object pose with an updated object pose based on the updated image pose and the input requesting manipulation of the virtual object.

13. The system of claim 12, wherein the computer readable code is further executable by the one or more processors to:

receive the reference image and a reference pose of the reference image from the reference database; and

in response to determining that the real object depicted in the image matches the real object depicted in the reference image, generate the updated image pose based in part on the reference pose.

14. The system of claim 12, further comprising a user interface device configured to receive the input requesting manipulation of the virtual object.

15. The system of claim 12, wherein the reference coordinate system is a geographic global coordinate system or a coordinate system associated with the real object depicted in the image.

16. The system of claim 12, wherein the computer readable code is further executable by the one or more processors to receive, from the server device, a modified version of the updated object pose.

17. The system of claim 12, further comprising a vehicle that includes the one or more processors.