Method for representing virtual information in a view of a real environment

• US 10,453,267 B2
• Filed: 09/10/2018
• Issued: 10/22/2019
• Est. Priority Date: 10/12/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Meier, Peter, Kuhn, Michael, Angermann, Frank
• Primary Examiner(s): Cobb, Michael J
• Application Number:
  US16/126,785
• Publication Number:
  US 20190005734A1
• Time in Patent Office: 407 Days
• Field of Search: None
• CPC Class Codes: G01S 19/01 Satellite radio beacon positioning systems transmitting time-stamped messages, e.g. GPS [Global Positioning System], GLONASS [Global Orbiting Navigation Satellite System] or GALILEO
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

A method for representing virtual information in a view of a real environment comprises providing a virtual object having a global position and orientation with respect to a geographic global coordinate system, with first pose data on the global position and orientation of the virtual object, in a database of a server, taking an image of a real environment by a mobile device and providing second pose data as to at which position and with which orientation with respect to the geographic global coordinate system the image was taken. The method further includes displaying the image on a display of the mobile device, accessing the virtual object in the database and positioning the virtual object in the image on the basis of the first and second pose data, manipulating the virtual object or adding a further virtual object, and providing the manipulated virtual object with modified first pose data or the further virtual object with third pose data in the database.

Claims

1. A method comprising:

determining an image pose of an image in a reference coordinate system;

receiving, from a computing device, a reference image and a reference pose of the reference image in the reference coordinate system;

matching features of a real object depicted in the image to corresponding features of a real object depicted in the reference image;

generating an updated image pose based at least in part on the reference pose in response to determining that the features of the real object depicted in the image match the corresponding features of the real object depicted in the reference image;

determining an overlay position of a virtual object in the image based on user input; and

sending, to the computing device, a request to store an object pose of the virtual object in the reference coordinate system based on the updated image pose and the overlay position.

2. The method of claim 1, further comprising:

capturing the image at a camera of a mobile device; and

receiving global positioning system (GPS) data from a GPS sensor of the mobile device, wherein the image pose is determined based on the GPS data.

3. The method of claim 1, further comprising:

capturing the image at a camera of a mobile device; and

receiving an identifier from a wireless local area network adapter of the mobile device, wherein the image pose is determined based on the identifier.

4. The method of claim 1, wherein the reference coordinate system is a geographic global coordinate system.

5. The method of claim 1, wherein the reference coordinate system is a coordinate system associated with the real object depicted in the image.

6. The method of claim 1, further comprising:

receiving a second object pose of a second virtual object from the computing device; and

displaying the second virtual object at a second overlay position determined based on the second object pose and the updated image pose.

7. A computer readable memory device comprising computer readable code executable by one or more processors to:

initiate capture of an image by a camera of a device;

determine an image pose of the image in a reference coordinate system;

receive, from a computing device, a reference image and a reference pose of the reference image in the reference coordinate system;

match features of a real object depicted in the image to corresponding features of a real object depicted in the reference image;

generate an updated image pose based at least in part on the reference pose in response to determining that the features of the real object depicted in the image match the corresponding features of the real object depicted in the reference image;

determine an overlay position of a virtual object in the image based on user input; and

initiate transmission, to the computing device, of a request to store an object pose of the virtual object in the reference coordinate system based on the updated image pose and the overlay position.

8. The computer readable memory device of claim 7, wherein the computer readable code is further executable by one or more processors to receive global positioning system (GPS) data from a GPS sensor of the device, wherein the image pose is determined based on the GPS data.

9. The computer readable memory device of claim 7, wherein the computer readable code is further executable by one or more processors to receive an identifier from a wireless local area network adapter of the device, wherein the image pose is determined based on the identifier.

10. The computer readable memory device of claim 7, wherein the reference coordinate system is a geographic global coordinate system.

11. The computer readable memory device of claim 7, wherein the reference coordinate system is a coordinate system associated with the real object depicted in the image.

12. The computer readable memory device of claim 7, wherein the computer readable code is further executable by one or more processors to:

receive a second object pose of a second virtual object from the computing device; and

initiate display of the second virtual object at a second overlay position determined based on the second object pose and the updated image pose.

13. An apparatus comprising:

a location sensor;

a camera;

one or more processors; and

a memory storing computer readable code executable by one or more processors to;

initiate capture of an image by the camera;

determine an image pose of the image in a reference coordinate system based on location data from the location sensor;

receive, from a computing device, a reference image and a reference pose of the reference image in the reference coordinate system;

match features of a real object depicted in the image to corresponding features of a real object depicted in the reference image;

generate an updated image pose based at least in part on the reference pose in response to determining that the features of the real object depicted in the image match the corresponding features of the real object depicted in the reference image;

determine an overlay position of a virtual object in the image based on user input; and

initiate transmission, to the computing device, of a request to store an object pose of the virtual object in the reference coordinate system based on the updated image pose and the overlay position.

14. The apparatus of claim 13, wherein the location sensor includes a global positioning system (GPS) sensor and the location data corresponds to GPS data.

15. The apparatus of claim 13, wherein the location sensor includes a wireless local area network adapter and the location data corresponds to an identifier of a wireless access point.

16. The apparatus of claim 13, wherein the reference coordinate system is a geographic global coordinate system.

17. The apparatus of claim 13, wherein the reference coordinate system is a coordinate system associated with the real object depicted in the image.

18. The apparatus of claim 13, further comprising a display device, wherein the computer readable code is further executable by one or more processors to initiate display, via the display device, of the virtual object at the overlay position within the image.

19. The apparatus of claim 18, wherein the display device includes a head mounted display device.

20. The apparatus of claim 18, wherein the computer readable code is further executable by one or more processors to:

receive a second object pose of a second virtual object from the computing device; and

initiate display, via the display device, of the second virtual object at a second overlay position determined based on the second object pose and the updated image pose.