METHOD FOR REPRESENTING VIRTUAL INFORMATION IN A VIEW OF A REAL ENVIRONMENT

• US 20150154803A1
• Filed: 02/06/2015
• Published: 06/04/2015
• Est. Priority Date: 10/12/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Metaio GmbH (Apple Inc.)
• Inventors: Meier, Peter, Kuhn, Michael, Angermann, Frank
• Granted Patent: US 10,074,215 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G01S 19/01 Satellite radio beacon positioning systems transmitting time-stamped messages, e.g. GPS [Global Positioning System], GLONASS [Global Orbiting Navigation Satellite System] or GALILEO
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

A method for representing virtual information in a view of a real environment comprises providing a virtual object having a global position and orientation with respect to a geographic global coordinate system, with first pose data on the global position and orientation of the virtual object, in a database of a server, taking an image of a real environment by a mobile device and providing second pose data as to at which position and with which orientation with respect to the geographic global coordinate system the image was taken. The method further includes displaying the image on a display of the mobile device, accessing the virtual object in the database and positioning the virtual object in the image on the basis of the first and second pose data, manipulating the virtual object or adding a further virtual object, and providing the manipulated virtual object with modified first pose data or the further virtual object with third pose data in the database.

Claims

1. A method for representing image information in a view of a real environment, comprising the steps of:

1a) taking a first plurality of images of a real environment by means of at least one first mobile device;

1b) storing the first plurality of images in at least one database;

1c) taking a second image of the real environment by means of a second mobile device;

1d) accessing the first plurality of images stored in the at least one database;

1e) matching the first plurality of images against the first plurality of images wherein said matching comprises performing bundle adjustment;

1f) determining, for each respective image of the first plurality of images, first pose data permitting a conclusion to be made as to a position and an orientation with respect to a reference coordinate system in which the respective image was taken according to a result of said matching;

1 g) comparing the second image with at least one of the first plurality of images;

1h) determining, for at least one presentation image among the first plurality of images, an overlay position in the second image according to a result of said matching and the first pose data associated with the at least one presentation image; and

1i) displaying the at least one presentation image overlaid at the determine overlay position in the second image on a display device.

2. The method of claim 1, comprising the further steps of:

providing initial values for the first pose data associated with at least one of the first plurality of images;

and wherein, in the step 1e), said bundle adjustment is performed with consideration of the at least one first pose data.

3. The method of claim 1, comprising the further steps of:

determining second pose data permitting a conclusion to be made as to a position and an orientation with respect to the reference coordinate system in which the second image was taken according to a result of said comparing.

4. The method of claim 1, comprising the further steps of:

providing second pose data permitting a conclusion to be made as to a position and an orientation with respect to the reference coordinate system in which the second image was taken; and

wherein, in the step 1g), said comparing is performed with consideration of the second pose data and the second pose data is modified according to a result of said comparing.

5. The method of claim 4, wherein the overlay position is determined according to the modified second pose data.

6. The method of claim 1, wherein at least one of the steps 1d), 1e), 1f), 1g), 1h) is executed in the second mobile device or in at least one server.

7. The method of claim 1, comprising the further steps of:

providing a reference database with reference views of a real environment together with pose data permitting a conclusion to be made as to a position and an orientation with respect to the reference global coordinate system in which the respective reference view was taken by a camera;

comparing at least one real object that is shown in the second image with at least part of a real object that is contained in at least one of the reference views, and matching of the second pose data of the image with the pose data of the at least one reference view; and

modifying at least part of the second pose data on the basis of at least part of the pose data of the at least one reference view as a result of said matching.

8. The method of claim 1, wherein said matching in the step 1e) comprises matching image features of each of the first plurality of images with corresponding features of other images of the first plurality of images;

and

said comparing in the step 1g) comprises matching image features of the at least one of the first plurality of images with corresponding features of the second image.

9. The method of claim 1, wherein the second mobile device comprises, or is connected to, the display device.

10. The method of claim 1, wherein the reference coordinate system is a geographic global coordinate system or a coordinate system associated with a real object in the real environment.

11. The method of claim 1, wherein the at least one first mobile device and the second mobile device are the same client device, and wherein the at least one database comprises, or is connected to, the client device or at least one server.

12. The method of claim 1, wherein the first mobile device and the second mobile device are different client devices, and the at least one database comprises or is connected to at least one server.

13. A method for representing virtual information in a view of a real environment, comprising the steps of:

13a) taking at least one first image of a real environment by means of at least one first mobile device and providing first pose data permitting a conclusion to be made as to a position and an orientation with respect to a reference coordinate system where the at least one first image was taken;

13b) adding at least one virtual object by corresponding positioning in the at least one first image;

13c) providing second pose data in accordance with the positioning of the at least one virtual object in the at least one first image, wherein the second pose data permits a conclusion to be made as to a position and an orientation of the at least one virtual object with respect to the reference coordinate system;

13d) storing the at least one virtual object, the at least one first image, the first pose data and the second pose data in at least one database;

13e) taking a second image of the real environment by a second mobile device;

13f) accessing the at least one virtual object, the at least one first image, the first pose data and the second pose data stored in the at least one database;

13g) comparing at least one real object that is shown in the at least one first image with at least part of a real object that is contained in the second image;

13h) determining third pose data according to a result of comparison, wherein the third pose data permits a conclusion to be made as to a position and an orientation with respect to the reference coordinate system where the second image was taken; and

13i) displaying the at least one virtual object superimposed with the second image on a display device on the basis of the second pose data and the third pose data.

14. The method of claim 13, wherein the second mobile device comprises, or is connected to, the display device.

15. The method of claim 13, wherein the reference coordinate system is a geographic global coordinate system or a coordinate system associated with a real object in the real environment.

16. The method of claim 13, wherein the at least one first mobile device and the second mobile device are the same client device, and wherein the at least one database comprises, or is connected to, the client device or at least one server.

17. The method of claim 13, wherein the first mobile device and the second mobile device are different client devices, and the at least one database comprises or is connected to at least one server.

18. The method of claim 13, wherein at least one of the steps 13f), 13g), and 13h) is executed in the second mobile device or in at least one server.

19. The method of claim 13, wherein said comparing in the step 13g) comprises matching image features of the at least one first image with corresponding features of the second image.

20. A method for representing virtual information in a view of a real environment, comprising the steps of:

20a) providing at least one first view of a real environment by means of first data glasses together with first pose data permitting a conclusion to be made as to a position and an orientation with respect to a reference coordinate system where the first data glasses are positioned;

20b) providing at least one first image of the real environment taken by a camera connected to the first data glasses;

20c) adding at least one virtual object by corresponding positioning in the at least one first view of the real environment;

20d) providing second pose data in accordance with the positioning of the at least one virtual object in the at least one first view of the real environment, wherein the second pose data permits a conclusion to be made as to a position and an orientation of the at least one virtual object with respect to the reference coordinate system;

20e) storing the at least one virtual object, the at least one first image, the first pose data, and the second pose data in at least one database;

20f) providing at least one second view of a real environment by means of second data glasses;

20g) providing at least one second image of the real environment taken by a camera connected to the second data glasses;

20h) accessing the at least one virtual object, the at least one first image, the first pose data, and the second pose data stored in the at least one database;

20i) comparing at least one real object that is shown in the at least one first image with at least part of a real object that is contained in the at least one second image;

20j) determining third pose data according to a result of comparison, wherein the third pose data permits a conclusion to be made as to a position and an orientation with respect to the reference coordinate system where the second data glasses are positioned; and

20k) displaying the at least one virtual object superimposed with the at least one second view on the basis of the second pose data and the third pose data.

21. The method of claim 20, wherein the first pose data is determined according to the at least one first image.

22. The method of claim 20, wherein the first data glasses and the second data glasses are the same client device or different client devices.