Augmented reality on wireless mobile devices

• US 9,129,429 B2
• Filed: 03/15/2013
• Issued: 09/08/2015
• Est. Priority Date: 10/24/2012
• Status: Active Grant

• Current Assignee: Elbit Systems of America LLC (Elbit Systems Limited)
• Original Assignee: Exelis Incorporated (L3Harris Technologies, Inc.)
• Inventors: Hammond, John B.
• Primary Examiner(s): Tung, Kee M
• Assistant Examiner(s): Li, Grace Q
• Application Number:
  US13/837,902
• Publication Number:
  US 20140111544A1
• Time in Patent Office: 907 Days
• Field of Search: 345/633
• US Class Current: 1/1
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G09G 2370/16 Use of wireless transmission of display information
• CPC Class Codes: G09G 5/003 Details of a display terminal, the details relating to the control arrangement of the display terminal and to the interfaces thereto specific for a CRT G09G1/165; for a flat panel G09G3/2092

Abstract

An augmented reality control system in which the generation and utilization of simple geometric forms of real world objects at the enterprise level are layered onto or based on real world objects. Digital data is aligned to real world objects via surrogate of simplified geometric forms.

Claims

1. A mobile device configured for augmented reality comprising:

an image capture device operable to capture an image of a real scene;

a location sensor operable to determine the geographic location of the mobile device;

an orientation sensor operable to determine the orientation of the mobile device; and

a processor operable to;

determine a pointing vector directed towards the real scene from the orientation of the mobile device;

receive wireframe geometry data over a wireless communication link from an enterprise server storing digital data defined on a digital coordinate system, the received wireframe geometry data corresponding to a predetermined area surrounding the geographic location of the mobile device;

filter the received wireframe geometry data to a region aligned on the pointing vector, the region being smaller than the area surrounding the geographic location of the mobile device; and

register the filtered geometry data onto the image.

2. The mobile device of claim 1, wherein the orientation sensor is a sensor suite including a compass and an inertial measurement unit.

3. The mobile device of claim 1, wherein the location sensor is a global positioning system receiver.

4. The mobile device of claim 1, further comprising a head-up display on which the wireframe geometry data is displayed.

5. The mobile device of claim 1, wherein the processor is configured to perform hidden line removal and perspective setting prior to registering the filtered geometry data onto the image.

6. The mobile device of claim 1, wherein the processor is further operable to register the wireframe geometry data by matching wireframe features defined on the digital data coordinates in the received geometry data with edges in the image captured by the image capture device.

7. The mobile device of claim 1, wherein the processor is further operable to:

filter the received wireframe geometry data, in response to affirming that the pointing vector is moved outside the filtered geometry data, to another region aligned on a new pointing vector and defined by the extent of a new image captured by the image capture device.

8. The mobile device of claim 1, wherein the processor is further operable to:

determine whether the pointing vector is moved outside the received wireframe geometry data;

receive other wireframe geometry data from the enterprise server for the digital coordinates corresponding to the predetermined area surrounding the geographic location of the mobile device; and

filter the received other wireframe geometry data to another region aligned on another pointing vector.

9. The mobile device of claim 1, wherein the wireless communication link between the mobile device and the enterprise server includes a cellular network wireless communication link.

10. The mobile device of claim 1, wherein the wireless communication link between the mobile device and the enterprise server includes an IEEE 802.11 compliant wireless communication link.

11. The mobile device of claim 1, wherein the enterprise server establishes the region around the location of the mobile device as a two (2) kilometer diameter region in the digital coordinate system.

12. A method of augmenting a real scene captured by a mobile device, the method comprising:

determining a geographic location of the mobile device via a location sensor;

determining a pointing vector of the mobile device via an orientation sensor;

transmitting the geographic location and the pointing vector of the mobile device from the mobile device to an enterprise server over a wireless communication link established therebetween, the enterprise server storing digital data defined on a digital coordinate system and wireframe geometry data representative of the digital data defined on the digital coordinate system;

receiving the wireframe geometry data from the enterprise server over the wireless communication link, the received wireframe geometry data corresponding to, in the digital coordinate system, an area surrounding the geographic location of the mobile device;

filtering the wireframe geometry data to a region aligned on the pointing vector of the mobile device, the region being smaller than the area surrounding the geographic location of mobile device;

capturing a video frame of the real scene aligned on the pointing vector; and

registering the filtered wireframe geometry data onto the captured video frame.

13. The method according to claim 12 further comprising:

comparing the filtered wireframe geometry data with the device location; and

receiving updated wireframe geometry data from the enterprise server only upon affirming that the geographic location of the mobile device is outside the area surrounding the geographic location of the mobile device in the digital coordinate system.

14. The method of claim 12, further comprising:

selecting, through a user interface by a user, digital data from a data source;

aligning the digital data on a user selected location in the digital coordinate system;

receiving the digital data in the wireframe geometry data over the wireless communication link; and

registering the digital data on the image.

15. The method of claim 12, further comprising:

filtering the received wireframe geometry data, in response to affirming that the pointing vector is moved outside the filtered wireframe geometry data, to another region aligned on another pointing vector.

16. The method of claim 12, further comprising:

determining whether the pointing vector is moved outside the received wireframe geometry data;

receiving other wireframe geometry data from the enterprise server for the digital coordinates corresponding to the predetermined area surrounding the geographic location of the mobile device; and

filtering the received wireframe geometry data to another region aligned on another pointing vector.

17. The method of claim 12, further comprising:

organizing the wireframe geometry data, at the enterprise server, in accordance with proximity in the area surrounding the geographic location of the mobile device in the digital coordinate system to the pointing vector in the digital coordinate system.

18. A method of augmenting a real scene captured by a mobile device, the method comprising:

determining a geographic location of the mobile device via a location sensor;

capturing at least one video frame of the real scene and generating a corresponding at least one image from the video frame;

determining a pointing vector of the mobile device via an orientation sensor;

transmitting the image, pointing vector, and the geographic location of the mobile device from the mobile device to an enterprise server over a wireless communication link established therebetween, the enterprise server storing digital data defined on a digital coordinate system and wireframe geometry data representative of the digital data defined on the digital coordinate system;

filtering, at the enterprise server, the wireframe geometry data to a region aligned on the pointing vector at the geographic location;

registering, at the enterprise server, the filtered wireframe geometry data on the image and determining an initial coordinate therefrom;

transmitting the wireframe geometry data and the initial coordinate from the enterprise server to the mobile device over the wireless communication link for all of the at least one image on which the filtered wireframe geometry data has been registered; and

overlaying, at the mobile device, the wireframe geometry data received from the enterprise server onto all of the at least one image using the initial coordinate.

19. The method of claim 18, further comprising:

capturing a plurality of the video frames of the real scene and generating a corresponding plurality of images from the video frames;

transmitting the images with the pointing vector and the geographic location of the mobile device to the enterprise server;

registering, at the enterprise server, the filtered wireframe data onto the images and determining therefrom respective initial coordinates for the images; and

overlaying, at the mobile device, the wireframe geometry data received from the geometry data onto the images using the respective initial coordinates.

20. The method of claim 19, further comprising:

organizing the wireframe geometry data received by the mobile device in accordance with proximity to the pointing vector of a predetermined one of the images;

determining whether the pointing vector has changed from the real scene corresponding to the predetermined one of the images; and

selecting other wireframe geometry data received by the mobile device corresponding to the real scene toward which the changed pointing vector is directed.