Augmented reality for wireless mobile devices

• US 10,055,890 B2
• Filed: 09/15/2016
• Issued: 08/21/2018
• Est. Priority Date: 10/24/2012
• Status: Active Grant

• Current Assignee: Elbit Systems of America LLC (Elbit Systems Limited)
• Original Assignee: Harris Corporation (L3Harris Technologies, Inc.)
• Inventors: Hammond, John B.
• Primary Examiner(s): Tung, Kee M
• Assistant Examiner(s): Li, Grace Q
• Application Number:
  US15/266,136
• Publication Number:
  US 20170004658A1
• Time in Patent Office: 705 Days
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G09G 2370/16 Use of wireless transmission of display information
• CPC Class Codes: G09G 5/003 Details of a display terminal, the details relating to the control arrangement of the display terminal and to the interfaces thereto specific for a CRT G09G1/165; for a flat panel G09G3/2092

Abstract

A model includes model layers on which a wireframe representation of objects located at geographic coordinates is stored in a memory such that surface detail of the objects increases from a base model layer to an uppermost model layer. Digital data layers stored in the memory that encompass digital coordinates corresponding with the geographic coordinates. Digital content for augmenting scenes is stored on pre-selected digital data layers at pre-selected digital coordinates on those layers. One or more of the digital data layers are logically linked with one or more of the model layers. When the location and spatial orientation of a mobile device in which a scene is viewed is received, the digital content on the digital data layer logically linked to one of the model layers is transmitted over a wireless communication channel to the mobile device.

Claims

1. A method for reality augmentation comprising:

storing, in a memory device;

a model representing real world objects located at corresponding geographic coordinates of a geographic coordinate system, the model comprising a data structure of model layers in which a wireframe representation of the real world objects is represented in a base model layer and in which surface detail of the real world objects increases from the base model layer to an uppermost model layer;

a data structure having a plurality of digital data layers encompassing digital coordinates of a digital coordinate system corresponding with respective geographic coordinates of the geographic coordinate system; and

digital content on pre-selected one or more of the digital data layers at pre-selected digital coordinates thereon, the digital content comprising data for augmenting scenes containing the real world objects at the respective locations;

logically linking one or more of the digital data layers with one or more of the model layers to locate the digital content within the scenes based upon the digital coordinates and the model;

receiving the location and spatial orientation of a mobile device in which a scene is viewed by the mobile device that overlaps with the geographic coordinates; and

transmitting, over a wireless communication channel to the mobile device, the digital content on the digital data layer logically linked to a selected one of the model layers for geographic coordinates that include the scene viewed by the mobile device to augment the scene viewed by the mobile device using the digital content.

2. The method of claim 1, wherein transmitting the digital content further comprises:

transmitting the digital content on the base model layer as the selected model layer and the wireframe representation of the real world objects in the scene viewed by the mobile device.

3. The method of claim 2, wherein transmitting the digital content further comprises;

transmitting the digital content and the wireframe representation of the real world objects in a predetermined area surrounding the location of the mobile device.

4. The method of claim 1, further comprising:

identifying key features in image data of the real world objects, each of the key features being located at known geographic coordinates of the geographic coordinate system; and

constructing the wireframe representation of the image data from data points at the digital coordinates of the digital coordinate system corresponding to the known geographic coordinates of identified key features; and

storing the wireframe model on the base model layer.

5. The method of claim 1, further comprising:

retrieving the base model layer corresponding with image data presented in the view of the mobile device in response to receiving the image data and the location of the mobile device;

registering the wireframe representation of the retrieved base model layer on the received image data; and

transmitting the digital content at the digital coordinates on the digital data layer corresponding with the geographic coordinates of the registered image data.

6. The method of claim 1, further comprising:

determining bandwidth of the wireless communication channel; and

transmitting, over the wireless communication channel, one of the model layers having greater surface detail than the wireframe representation on the base model layer with the digital content on the digital data layer logically linked therewith based on the determined bandwidth.

7. A server apparatus for augmenting reality comprising:

a memory device configured to store;

a model representing real world objects located at corresponding geographic coordinates of a geographic coordinate system, the model comprising a data structure of model layers in which a wireframe representation of the real world objects is represented in a base model layer and in which surface detail of the real world objects increases from the base model layer to an uppermost model layer;

a data structure having a plurality of digital data layers encompassing digital coordinates of a digital coordinate system corresponding with respective geographic coordinates of the geographic coordinate system; and

digital content on pre-selected one or more of the digital data layers at pre-selected digital coordinates thereon, the digital content comprising data for augmenting scenes containing the real world objects at the respective locations; and

a processor configured to;

logically link one or more of the digital data layers with one or more of the model layers to locate the digital content within the scenes based upon the digital coordinates and the model;

receive the location and spatial orientation of a mobile device in which a scene is viewed by the mobile device that overlaps with the geographic coordinates; and

transmit, over a wireless communication channel to the mobile device, the digital content on the digital data layer logically linked to a selected one of the model layers for geographic coordinates that include the scene viewed by the mobile device to augment the scene viewed by the mobile device using the digital content.

8. The server apparatus of claim 7, wherein the processor is further configured to:

transmit the digital content on the base model layer as the selected model layer and the wireframe representation of the real world objects in the scene viewed by the mobile device.

9. The server apparatus of claim 8, wherein the processor is further configured to:

transmit the digital content and the wireframe representation of the real world objects in a predetermined area surrounding the location of the mobile device.

10. The server apparatus of claim 7, wherein the processor is further configured to:

identify key features in image data of the real world objects, each of the key features being located at known geographic coordinates of the geographic coordinate system; and

construct the wireframe representation of the image data from data points at the digital coordinates of the digital coordinate system corresponding to the known geographic coordinates of identified key features; and

store the wireframe model on the base model layer.

11. The server apparatus of claim 7, wherein the processor is further configured to:

retrieve the base model layer corresponding with image data presented in the view of the mobile device in response to receiving the image data and the location of the mobile device;

register the wireframe representation of the retrieved base model layer on the received image data; and

transmit the digital content at the digital coordinates on the digital data layer corresponding with the geographic coordinates of the registered image data.

12. The server apparatus of claim 7, wherein the processor is further configured to:

determine bandwidth of the wireless communication channel; and

transmit, over the wireless communication channel, one of the model layers having greater surface detail than the wireframe representation on the base model layer with the digital content on the digital data layer logically linked therewith based on the determined bandwidth.

13. A non-transitory, tangible computer-readable medium having stored thereon processor instructions that, when executed by a processor, causes the processor to:

store, in a memory device;

a model representing real world objects located at corresponding geographic coordinates of a geographic coordinate system, the model comprising a data structure of model layers in which a wireframe representation of the real world objects is represented in a base model layer and in which surface detail of the real world objects increases from the base model layer to an uppermost model layer;

a data structure having a plurality of digital data layers encompassing digital coordinates of a digital coordinate system corresponding with respective geographic coordinates of the geographic coordinate system; and

digital content on pre-selected one or more of the digital data layers at pre-selected digital coordinates thereon, the digital content comprising data for augmenting scenes containing the real world objects at the respective locations;

logically link one or more of the digital data layers with one or more of the model layers to locate the digital content within the scenes based upon the digital coordinates and the model;

receive the location and spatial orientation of a mobile device in which a scene is viewed by the mobile device that overlaps with the geographic coordinates; and

transmit, over a wireless communication channel to the mobile device, the digital content on the digital data layer logically linked to a selected one of the model layers for geographic coordinates that include the scene viewed by the mobile device to augment the scene viewed by the mobile device using the digital content.

14. The computer-readable medium of claim 13, wherein the processor instructions include processor instructions that cause the processor to:

transmit the digital content on the base model layer as the selected model layer and the wireframe representation of the real world objects in the scene viewed by the mobile device.

15. The computer-readable medium of claim 14, wherein the processor instructions include processor instructions that cause the processor to:

transmit the digital content and the wireframe representation of the real world objects in a predetermined area surrounding the location of the mobile device.

16. The computer-readable medium of claim 13, wherein the processor instructions include processor instructions that cause the processor to:

identify key features in image data of the real world objects, each of the key features being located at known geographic coordinates of the geographic coordinate system; and

construct the wireframe representation of the image data from data points at the digital coordinates of the digital coordinate system corresponding to the known geographic coordinates of identified key features; and

store the wireframe model on the base model layer.

17. The computer-readable medium of claim 13, wherein the processor instructions include processor instructions that cause the processor to:

retrieve the base model layer corresponding with image data presented in the view of the mobile device in response to receiving the image data and the location of the mobile device;

register the wireframe representation of the retrieved base model layer on the received image data; and

transmit the digital content at the digital coordinates on the digital data layer corresponding with the geographic coordinates of the registered image data.

18. The computer-readable medium of claim 13, wherein the processor instructions include processor instructions that cause the processor to:

determine bandwidth of the wireless communication channel; and

transmit, over the wireless communication channel, one of the model layers having greater surface detail than the wireframe representation on the base model layer with the digital content on the digital data layer logically linked therewith based on the determined bandwidth.