Network-based real time registered augmented reality for mobile devices

• US 9,240,074 B2
• Filed: 10/06/2011
• Issued: 01/19/2016
• Est. Priority Date: 10/10/2010
• Status: Active Grant

• Current Assignee: Rafael Advanced Defense Systems Limited (Government of Israel)
• Original Assignee: Rafael Advanced Defense Systems Limited (Government of Israel)
• Inventors: Berkovich, Erez, Littwitz, Elon, Adler, Avigdor
• Primary Examiner(s): Chu, David H
• Application Number:
  US13/822,652
• Publication Number:
  US 20130187952A1
• Time in Patent Office: 1,566 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 1/00244 with a server, e.g. an internet server fax-servers or the like for store and forward H04N1/324
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/32101 Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title
• CPC Class Codes: H04N 1/32144 embedded in the image data, i.e. enclosed or integrated in the image, e.g. watermark, super-imposed logo or stamp
• CPC Class Codes: H04N 2201/0084 Digital still camera
• CPC Class Codes: H04N 2201/3245 of image modifying data, e.g. handwritten addenda, highlights or augmented reality information
• CPC Class Codes: H04N 2201/3253 Position information, e.g. geographical position at time of capture, GPS data
• CPC Class Codes: H04N 2201/3273 Display
• CPC Class Codes: H04N 2201/3278 Transmission
• CPC Class Codes: H04N 23/633 for displaying additional information relating to control or operation of the camera
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04W 4/02 Services making use of location information
• CPC Class Codes: H04W 4/029 Location-based management or tracking services
• CPC Class Codes: H04W 4/80 Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication
• CPC Class Codes: H04W 64/00 Locating users or terminals or network equipment for network management purposes, e.g. mobility management
• CPC Class Codes: H04W 88/02 Terminal devices

Abstract

A method of operating a mobile device with a camera, a display and a position sensor to provide a display of supplementary information aligned with a view of a scene. One or more image obtained from the camera is uploaded to a remote server together with corresponding data from the position sensor. Image processing is then performed to track image motion between that image and subsequent images obtained from the camera, determining a mapping between the uploaded image and a current image. Data is then received via the network indicative of a pixel location for display of supplementary information within the reference image. The mapping is used to determine a corresponding pixel location for display of the supplementary information within the current image, and the supplementary information is displayed on the display correctly aligned with the view of the scene.

Claims

1. A method of operating a mobile device to provide a display of supplementary information aligned with a view of a scene, the mobile device including a camera, a display, at least one processor and at least one position sensor, the mobile device being in networked communication via a mobile network with a remote server, the method comprising the steps of:

(a) obtaining a first image from the camera;

(b) uploading at least said first image via the mobile network to the remote server together with corresponding data from the at least one position sensor;

(c) performing image processing to track image motion between said first image and subsequent images obtained from the camera, and hence determining a mapping between features of a reference image and features of a current image obtained from the camera, said reference image being selected from the group consisting of;

said first image; and

a second image for which transformation parameters were uploaded to the server corresponding to a mapping between said first image and said second image, said processing being performed by the at least one processor of the mobile device;

(d) receiving via the mobile network data indicative of a pixel location for display of supplementary information within said reference image, said receiving including receiving data indicative of a pixel location for display of supplementary information relating to an object obscured from view;

(e) employing said mapping to determine a corresponding pixel location for display of the supplementary information within the current image; and

(f) displaying the supplementary information on the display correctly aligned with the view of the scene.

2. The method of claim 1, wherein said mapping is a global spatial transformation approximating said image motion.

3. The method of claim 1, wherein said mapping is an optical flow mapping.

4. The method of claim 1, wherein said mapping is derived by simultaneous localization and mapping (SLAM) processing of said images.

5. The method of claim 1, wherein said displaying includes displaying the supplementary information overlying an image of the view of the scene on a non-transparent display.

6. The method of claim 1, wherein said displaying includes displaying the supplementary information on a transparent display aligned with a direct view of the scene.

7. The method of claim 1, wherein said receiving includes receiving data indicative of a graphic form for display of the supplementary information.

8. The method of claim 1, wherein said pixel location for display of supplementary information received by the mobile device is derived at least in part by matching of said first image to a three dimensional model of the scene.

9. The method of claim 1, wherein said image processing is continued after said displaying to generate an updated mapping for a new current image, and wherein said pixel location for display of the supplementary information within the current image is updated according to said updated mapping.

10. The method of claim 1, further comprising the steps of:

(a) inputting from a user a request for information regarding a pixel location in the current image;

(b) employing said mapping to derive a corresponding pixel location in said reference image; and

(c) transmitting to the server a request for information regarding the pixel location in said reference image.

11. The method of claim 10, wherein said request for information is a request for range information relating to a viewed location.

12. The method of claim 1, further comprising the steps of:

(a) evaluating a tracking reliability indicator based upon at least one parameter selected from the group consisting of;

elapsed time since sampling of said first image;

degree of overlap between the current image and said first image; and

number of tracked features in common between the current image and said first image; and

(b) when said tracking reliability indicator passes a threshold, uploading a new reference image obtained from the camera to the server.

13. A mobile device for providing a display of supplementary information aligned with a view of a scene, the mobile device comprising:

(a) a camera;

(b) a display;

(c) at least one position sensor;

(d) a communication module for networked communication to a remote server; and

(e) a processing system including at least one processor, said processing system being in data communication with said camera, said display, said at least one position sensor and said communication module, said processing system being configured to implement the method of claim 1.

14. The mobile device of claim 13, wherein the mobile device is implemented as a device selected from the group consisting of:

an augmented reality headset;

binoculars;

a telescope; and

night vision goggles.

15. The mobile device of claim 13, wherein the mobile device is implemented as a device selected from the group consisting of:

a cellular telephone;

a hand-held computer;

a navigation device;

a personal digital assistant; and

a digital camera.

16. A method of operating a mobile device to provide a display of supplementary information aligned with a view of a scene, the mobile device including a camera, a display, at least one processor and at least one position sensor, the mobile device being in networked communication via a mobile network with a remote server, the method comprising the steps of:

(a) obtaining a first image from the camera;

(b) uploading at least said first image via the mobile network to the remote server together with corresponding data from the at least one position sensor;

(c) performing image processing to track image motion between said first image and subsequent images obtained from the camera, and hence determining a mapping between features of a reference image and features of a current image obtained from the camera, said reference image being selected from the group consisting of;

said first image; and

a second image for which transformation parameters were uploaded to the server corresponding to a mapping between said first image and said second image, said processing being performed by the at least one processor of the mobile device;

(d) receiving via the mobile network data indicative of a pixel location for display of supplementary information within said reference image;

(e) employing said mapping to determine a corresponding pixel location for display of the supplementary information within the current image;

(f) displaying the supplementary information on the display correctly aligned with the view of the scene;

(g) evaluating a tracking reliability indicator based upon at least one parameter selected from the group consisting of;

elapsed time since sampling of said first image;

degree of overlap between the current image and said first image; and

number of tracked features in common between the current image and said first image; and

(h) when said tracking reliability indicator passes a threshold, uploading a new reference image obtained from the camera to the server.

17. A mobile device for providing a display of supplementary information aligned with a view of a scene, the mobile device comprising:

(a) a camera;

(b) a display;

(c) at least one position sensor;

(d) a communication module for networked communication to a remote server; and

(e) a processing system including at least one processor, said processing system being in data communication with said camera, said display, said at least one position sensor and said communication module, said processing system being configured to implement the method of claim 16.

18. The mobile device of claim 17, wherein the mobile device is implemented as a device selected from the group consisting of:

an augmented reality headset;

binoculars;

a telescope; and

night vision goggles.

19. The mobile device of claim 17, wherein the mobile device is implemented as a device selected from the group consisting of:

a cellular telephone;

a hand-held computer;

a navigation device;

a personal digital assistant; and

a digital camera.