MOBILE AUGMENTED REALITY SYSTEM

• US 20130002649A1
• Filed: 07/01/2011
• Published: 01/03/2013
• Est. Priority Date: 07/01/2011
• Status: Active Grant

• Current Assignee: Tahoe Research Limited (f/k/a Learndale Limited) (Vector Capital Corporation)
• Original Assignee: Intel Corporation
• Inventors: Wu, Yi, Takacs, Gabriel, Choubassi, Maha El, Kozintsev, Igor V.
• Granted Patent: US 9,600,933 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Embodiments of the invention relate to systems, apparatuses and methods to provide image data, augmented with related data, to be displayed on a mobile computing device. Embodiments of the invention display a live view augmented with information identifying an object amongst other objects. Embodiments of the invention may utilize other related data, such as 3D point cloud data, image data and location data related to the object, to obtain a specific location of an object within the live view. Embodiments of the invention may further display a live view with augmented data three-dimensionally consistent with the position and orientation of the image sensor of the mobile computing device.

Claims

1. An apparatus comprising:

a processor;

a memory; and

an augmentation module included in the memory and executed via the processor to receive data identifying a point-of-interest (POI),

detect a surface of a structure included in a live view captured by an image sensor, the structure to include the POI,

determine a position and an orientation of the image sensor relative to the POI based, at least in part, on the detected surface of the structure included in the live view and data from a second sensor,

track the position and the orientation of the image sensor while the image sensor captures the live view, and

generate display data including image data associated with the POI for augmenting the live view, wherein the image data is three-dimensionally consistent with the position and the orientation of the image sensor when displayed in the live view at the POI on the detected surface of the structure.

2. The apparatus of claim 1, wherein the apparatus comprises a server device operatively coupled to a mobile computing device, the mobile computing device to include the image sensor, the second sensor, and a display to display an augmented live view including the image data associated with the POI.

3. The apparatus of claim 1, wherein the apparatus comprises a mobile computing device, the mobile computing device to further include a display to display an augmented live view including the image data associated with the POI.

4. The apparatus of claim 1, wherein the structure comprises a building and the detected surface of the structure comprises an external side of the building.

5. The apparatus of claim 1, wherein the data from the second sensor comprises Global Position Satellite (GPS) data.

6. The apparatus of claim 3, further comprising:

a mobile phone operating unit, the second sensor to capture geographical information of the apparatus via cellular phone tower triangulation.

7. The apparatus of claim 1, wherein the data from the second sensor comprises acceleration data captured via an accelerometer.

8. The apparatus of claim 1, wherein the data from the second sensor comprises includes magnetic field data captured via a compass.

9. The apparatus of claim 1, the image data associated with the POI to identify a set of boundaries of the POI within the detected surface of the structure.

10. The apparatus of claim 1, the augmentation module to further determine a center of the POI within the detected surface of the structure, the display data to be generated based on the center of the POI.

11. The apparatus of claim 1, wherein the image data associated with the POI comprises at least one of static image data and video image data.

12. An article of manufacture comprising a non-transitory machine-readable storage medium that provides instructions that, if executed by the machine, will cause the machine to perform operations comprising:

receiving data identifying a point-of-interest (POI);

detecting a surface of a structure included in a live view captured by an image sensor, the structure to include the POI;

determining a position and an orientation of the image sensor relative to the POI based, at least in part, on the detected surface of the structure included in the live view and data from a second sensor;

tracking the position and the orientation of the image sensor while the image sensor captures the live view; and

generating display data including image data associated with the POI for augmenting the live view, wherein the image data is three-dimensionally consistent with the position and the orientation of the image sensor when displayed in the live view at the POI on the detected surface of the structure.

13. The article of manufacture of claim 12, wherein the structure comprises a building and the detected surface of the structure comprises an external side of the building.

14. The article of manufacture of claim 12, wherein the data from the second sensor comprises Global Position Satellite (GPS) data.

15. The article of manufacture of claim 12, wherein the machine comprises a mobile phone, and the data from the second sensor comprises geographical information of the mobile phone captured via cellular phone tower triangulation.

16. The article of manufacture of claim 12, wherein the data from the second sensor comprises acceleration data captured via an accelerometer.

17. The article of manufacture of claim 12, wherein the data from the second sensor comprises magnetic field data of captured via a compass.

18. The article of manufacture of claim 12, the image data associated with the POI to identify a set of boundaries of the POI within the detected surface of the structure.

19. The article of manufacture of claim 12, the operations further comprising determining a center of the POI within the surface of the structure, the display data generated based on the center of the POI.

20. The article of manufacture of claim 12, wherein the image data associated with the POI comprises at least one of static image data and video image data.

21. A method comprising:

receiving data identifying a point-of-interest (POI);

detecting a surface of a structure included in a live view captured by an image sensor, the structure to include the POI;

determining a position and an orientation of the image sensor relative to the POI based, at least in part, on the detected surface of the structure included in the live view and data from a second sensor;

tracking the position and the orientation of the image sensor while the image sensor captures the live view; and

generating display data including image data associated with the POI for augmenting the live view, wherein the image data is three-dimensionally consistent with the position and the orientation of the image sensor when displayed in the live view at the POI on the detected surface of the structure.

22. The method of claim 21, wherein the structure comprises a building and the detected surface of the structure comprises an external side of the building.

23. The method of claim 21, wherein the data from the second sensor comprises Global Position Satellite (GPS) data.

24. The method of claim 21, wherein the data from the second sensor comprises geographical information captured via cellular phone tower triangulation.

25. The method of claim 21, wherein the data from the second sensor comprises acceleration data captured via an accelerometer.

26. The method of claim 21, wherein the sensor data of the host device includes magnetic field data of the host device captured via a compass.

27. The method of claim 21, the image data associated with the POI to identify a set of boundaries of the POI within the detected surface of the structure.

28. The method of claim 21, further comprising determining a center of the POI within the surface of the structure, the display data to be generated based on the center of the POI.

29. The method of claim 21, wherein the image data associated with the POI comprises at least one of static image data and video image data.