METHOD FOR REPRESENTING VIRTUAL INFORMATION IN A VIEW OF A REAL ENVIRONMENT

• US 20120218263A1
• Filed: 10/11/2010
• Published: 08/30/2012
• Est. Priority Date: 10/12/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Metaio GmbH (Apple Inc.)
• Inventors: Meier, Peter, Kuhn, Michael, Angermann, Frank
• Granted Patent: US 9,001,154 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G01S 19/01 Satellite radio beacon positioning systems transmitting time-stamped messages, e.g. GPS [Global Positioning System], GLONASS [Global Orbiting Navigation Satellite System] or GALILEO
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

A method for representing virtual information in a view of a real environment comprises the following steps: providing at least one virtual object (10) having a global position and orientation with respect to a geographic global coordinate system (200), together with first pose data (PW10) permitting a conclusion to be made on the global position and orientation of the virtual object, in a database (3) of a server (2), taking at least one image (50) of a real environment (40) by means of a mobile device (30) and providing second pose data (PW50) permitting a conclusion to be made as to at which position and with which orientation with respect to the geographic global coordinate system (200) the image was taken, displaying the image (50) on a display (31) of the mobile device, accessing the virtual object (10) in the database (3) of the server (2) and positioning the virtual object (10) in the image (50) shown on the display on the basis of the first and second pose data (PW10, PW50), manipulating the virtual object (10) or adding a further virtual object (11) by corresponding positioning in the image (50) shown on the display, and providing the manipulated virtual object (10) together with modified first pose data (PW10) in accordance with the positioning in the image (50) or the further virtual object (11) together with third pose data in accordance with the positioning in the image (50) in the database (3) of the server (2), the modified first pose data and third pose data each permitting a conclusion to be made on the global position and orientation of the manipulated or further virtual object. Instead of with an image of the real environment, the method can also be performed analogously using a view of an HMD, for example.

Claims

1. A method for representing virtual information in a view of a real environment, comprising the steps of:

providing at least one virtual object having a global position and orientation with respect to a geographic global coordinate system, together with first pose data permitting a conclusion to be made on the global position and orientation of the virtual object, in a database of a server;

taking at least one image of a real environment by means of a mobile device and providing second pose data permitting a conclusion to be made as to at which position and with which orientation with respect to the geographic global coordinate system the image was taken;

displaying the image on a display of the mobile device;

accessing the virtual object in the database of the server and positioning the virtual object in the image shown on the display on the basis of the first and second pose data;

manipulating the virtual object or adding another virtual object by corresponding positioning in the image shown on the display; and

providing the manipulated virtual object together with modified first pose data in accordance with the positioning in the image or of the further virtual object together with third pose data in accordance with the positioning in the image in the database of the server, the modified first pose data and third pose data each permitting a conclusion to be made on the global position and orientation of the manipulated or further virtual object.

2. The method of claim 1, wherein the image is provided on the server together with the second pose data.

4. The method of claim 1, wherein the mobile device or the data glasses comprise, or are connected to, a means for generating the second pose data.

5. The method of claim 1, wherein the pose data each include respective three-dimensional values with respect to position and orientation.

6. The method of claim 1, wherein an orientation of the image of the real environment is defined independently of the earth'"'"'s surface.

7. The method of claim 1, wherein a storage location on the server stores in which image of several images of a real environment or in which view of several views of a real environment, which virtual object of several virtual objects has been provided with pose data.

8. The method of claim 1, comprising the further steps of:

providing a reference database with reference views of a real environment together with pose data permitting a conclusion to be made as to at which position and with which orientation with respect to the geographic global coordinate system the respective reference view was taken by a camera;

comparing at least one real object that is shown in the image with at least part of a real object that is contained in at least one of the reference views, and matching of the second pose data of the image with the pose data of the at least one reference view; and

modifying at least part of the second pose data on the basis of at least part of the pose data of the at least one reference view as a result of said matching.

9. The method of claim 8, wherein furthermore at least part of the first pose data of the virtual object positioned in the image is modified as a result of matching of the second pose data with the pose data of the at least one reference view.

10. The method of claim 1, comprising the further steps of:

providing the at least one image of the real environment together with the second pose data in the database of the server;

accessing the image of the real environment on the server and transmitting to a client device for displaying the image on the client device;

manipulating the virtual object or adding a further virtual object by corresponding positioning in the image of the real environment displayed on the client device;

providing the manipulated virtual object together with modified first pose data in accordance with the positioning in the image displayed on the client device or of the further virtual object together with third pose data in accordance with the positioning in the image displayed on the client device in the database of the server, the modified first pose data and third pose data each permitting a conclusion to be made on the global position and orientation of the manipulated or further virtual object in the image displayed on the client device.

11. The method of claim 10, further comprising the steps of:

accessing the image of the real environment on the server and transferring the same to a second client device to view the image on the second client device; and

accessing virtual objects provided on the server, with the view of the image on the second client device displaying those virtual objects the global position of which is within the real environment that is shown in the view of the image on the second client device.

12. The method of claim 1, further comprising the steps of:

providing a map view on the display of the mobile device; and

providing a choice for the user to select a viewing direction in taking the at least one image of the real environment.

13. The method of claim 1, comprising the steps of:

placing at least one virtual object in the image of the real environment and in a map view provided on a display of the mobile device; and

determining an orientation of the image of the real environment from a determined position of the image and the position of the at least one virtual object in the map view provided.

14. The method of claim 1, wherein in the event of failure of the network connection to the server, the data to be stored on the server are buffered on the mobile device and are transmitted to the server as soon as the network connection is available again.

15. The method of claim 1, wherein the user can retrieve a collection of scenes in a region of a real environment, which are made available to him for selection in a list sorted by proximity, or on a map or by means of augmented reality.

16. The method of claim 1, wherein the image or the virtual information has uniquely identifying characteristics, and an image or virtual information which already exists on a client or on the mobile device, is no longer downloaded from the server, but is loaded from a local data storage.

3. A method for representing virtual information in a view of a real environment, comprising the steps of:

providing at least one virtual object having a global position and orientation with respect to a geographic global coordinate system, together with first pose data permitting a conclusion to be made on the global position and orientation of the virtual object, in a database of a server;

providing at least one view of a real environment by means of data glasses together with second pose data permitting a conclusion to be made as to at which position and with which orientation with respect to the geographic global coordinate system the data glasses are positioned;

accessing the virtual object in the database of the server and positioning the virtual object in the view on the basis of the first and second pose data;

manipulating the virtual object or adding a further virtual object by corresponding positioning in the view; and

providing the manipulated virtual object together with modified first pose data in accordance with the positioning in the view or of the further virtual object together with third pose data in accordance with the positioning in the view in the database of the server, the modified first pose data and third pose data each permitting a conclusion to be made on the global position and orientation of the manipulated or further virtual object.