Mobile augmented reality system

• US 10,134,196 B2
• Filed: 03/21/2017
• Issued: 11/20/2018
• Est. Priority Date: 07/01/2011
• Status: Active Grant

• Current Assignee: Tahoe Research Limited (f/k/a Learndale Limited) (Vector Capital Corporation)
• Original Assignee: Intel Corporation
• Inventors: Wu, Yi, Takacs, Gabriel, El Choubassi, Maha, Kozintsev, Igor V.
• Primary Examiner(s): Chu, David H
• Application Number:
  US15/465,400
• Publication Number:
  US 20170337739A1
• Time in Patent Office: 609 Days
• Field of Search: None
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Embodiments of the invention relate to systems, apparatuses and methods to provide image data, augmented with related data, to be displayed on a mobile computing device. Embodiments of the invention display a live view augmented with information identifying an object amongst other objects. Embodiments of the invention may utilize other related data, such as 3D point cloud data, image data and location data related to the object, to obtain a specific location of an object within the live view. Embodiments of the invention may further display a live view with augmented data three-dimensionally consistent with the position and orientation of the image sensor of the mobile computing device.

Claims

1. An apparatus comprising:

a processor; and

a memory including instructions that, when executed by the processor, cause the processor to;

detect a surface of a structure included in a live view captured by an image sensor, the structure to include a point-of-interest (POI),

determine a position and an orientation of the image sensor relative to the POI based, at least in part, on the detected surface of the structure included in the live view and data from a second sensor,

extract, from the live view, a visual feature of the structure based on the position and the orientation, the visual feature associated with the POI,

track the position and the orientation of the image sensor while the image sensor captures the live view, and

generate display data including image data associated with the visual feature to augment the live view, the image data being three-dimensionally consistent with the position and the orientation of the image sensor when displayed in the live view at the POI on the detected surface of the structure.

2. The apparatus of claim 1, wherein the apparatus is a server in communication with a mobile computing device, the mobile computing device including the image sensor, the second sensor, and a display to display an augmented live view including the image data associated with the POI.

3. The apparatus of claim 1, wherein the apparatus is a mobile computing device, the mobile computing device including a display to display an augmented live view including the image data associated with the POI.

4. The apparatus of claim 1, wherein the detected surface of the structure includes an external side of a building.

5. The apparatus of claim 1, wherein the data from the second sensor includes Global Positioning System (GPS) data.

6. The apparatus of claim 1, further including:

a mobile phone, the second sensor to capture geographical information of the apparatus via cellular phone tower triangulation.

7. The apparatus of claim 1, wherein the data from the second sensor includes acceleration data captured via an accelerometer.

8. The apparatus of claim 1, wherein the data from the second sensor includes magnetic field data captured via a compass.

9. The apparatus of claim 1, wherein the image data associated with the POI is to indicate a set of boundaries of the POI within the detected surface of the structure.

10. The apparatus of claim 1, wherein the instructions further cause the processor to determine a center of the POI within the detected surface of the structure, and to generate the display data based on the center of the POI.

11. The apparatus of claim 1, wherein the image data associated with the POI includes at least one of static image data and video image data.

12. A memory comprising instructions that, when executed by at least one machine, cause the at least one machine to at least:

detect a surface of a structure included in a live view captured by an image sensor, the structure to include a point-of-interest (POI);

determine a position and an orientation of the image sensor relative to the POI based, at least in part, on the detected surface of the structure included in the live view and data from a second sensor;

extract, from the live view, a visual feature of the structure based on the position and the orientation, the visual feature associated with the POI;

track the position and the orientation of the image sensor while the image sensor captures the live view; and

generate display data including image data associated with the visual feature to augmenting the live view, the image data being three-dimensionally consistent with the position and the orientation of the image sensor when displayed in the live view at the POI on the detected surface of the structure.

13. The memory of claim 12, wherein the detected surface of the structure includes an external side of a building.

14. The memory of claim 12, wherein the data from the second sensor includes Global Positioning System (GPS) data.

15. The memory of claim 12, wherein the at least one machine includes a mobile phone, and the data from the second sensor includes geographical information of the mobile phone captured via cellular phone tower triangulation.

16. A method comprising:

accessing data identifying a point-of-interest (POI);

detecting, by executing an instruction with at least one processor, a surface of a structure included in a live view captured by an image sensor, the structure to include the POI;

determining, by executing an instruction with the at least one processor, a position and an orientation of the image sensor relative to the POI based, at least in part, on the detected surface of the structure included in the live view and data from a second sensor;

extracting, from the live view, a visual feature of the structure using the position and the orientation, the visual feature associated with the POI;

tracking the position and the orientation of the image sensor while the image sensor captures the live view; and

generating, by executing an instruction with the at least one processor, display data including image data associated with the visual feature for augmenting the live view, wherein the image data is three-dimensionally consistent with the position and the orientation of the image sensor when displayed in the live view at the POI on the detected surface of the structure.

17. The method of claim 16, wherein the detecting of the surface of the structure includes detecting an external side of a building.

18. The method of claim 16, further including accessing the data of the second sensor, the data from the second sensor including Global Positioning System (GPS) data.

19. The method of claim 16, further including accessing the data of the second sensor, the data from the second sensor including geographical information captured via cellular phone tower triangulation.

20. The method of claim 16, further including accessing the data of the second sensor, the data from the second sensor including acceleration data captured via an accelerometer.