METHODS, MOBILE DEVICE AND SERVER FOR SUPPORT OF AUGMENTED REALITY ON THE MOBILE DEVICE

• US 20130314443A1
• Filed: 05/14/2013
• Published: 11/28/2013
• Est. Priority Date: 05/28/2012
• Status: Abandoned Application

• Current Assignee: Clayton Grassick
• Original Assignee: Clayton Grassick
• Inventors: GRASSICK, Clayton
• Application Number:
  US13/893,811
• Publication Number:
  US 20130314443A1
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

The present disclosure relates to methods, to a mobile device and to a server for support of augmented reality on the mobile device. The mobile device acquires a current image of its environment. The mobile device determines its orientation and its location by comparing the current image with reference image information related to an estimated location of the mobile device. The mobile device may provide information about new, modified or missing reference image features, for a given location, to a server. The server may then update a corresponding reference image feature in a local database.

Claims

1. A method of determining a pose of a mobile device, comprising:

acquiring in the mobile device a current image of an environment of the mobile device; and

determining a location and an orientation of the mobile device by comparing the current image with reference image information related to an estimated location of the mobile device.

2. The method of claim 1, wherein the reference image information comprises a complete or partial reference image.

3. The method of claim 1, wherein the reference image information comprises features of one or more complete or partial reference images.

4. The method of claim 1, comprising:

detecting a new, modified or missing image feature in the current image; and

sending from the mobile device, toward a server, the new, modified or missing image feature.

5. The method of claim 1, wherein:

one or more sensors provide sensor information selected from the group consisting of an angular position of the mobile device relative to the horizon, an orientation of the mobile device relative to the magnetic North, an acceleration of the mobile device, an angular velocity of the mobile device, and a combination thereof; and

determining the location and the orientation of the mobile device further comprise refining the location and the orientation of the mobile device using the sensor information.

6. The method of claim 5, comprising applying the current image, the reference image information and the sensor information to a Kalman filter or other estimator.

7. The method of claim 1, comprising using a Kalman filter or other estimator to estimate pose using a comparison of the current image with reference image information.

8. The method of claim 1, comprising determining the location and the orientation of the mobile device in real-time.

9. The method of claim 1, comprising adding to the current image augmented reality information based on the determined location and the orientation of the mobile device.

10. A mobile device, comprising:

a camera for acquiring a current image of an environment of the mobile device; and

a processor for determining a location and an orientation of the mobile device by comparing the current image with reference image information related to an estimated location of the mobile device.

11. The mobile device of claim 10, wherein:

the processor is configured to detect a new, modified or missing image feature in the current image; and

the mobile device further comprises a transmitter for sending, toward a server, the new, modified or missing image feature.

12. The mobile device of claim 10, comprising:

one or more sensors for providing sensor information selected from the group consisting of an angular position of the mobile device relative to the horizon, an orientation of the mobile device relative to the magnetic North, an acceleration of the mobile device, an angular velocity of the mobile device and a combination thereof;

wherein the processor is configured refine the location and the orientation of the mobile device using the sensor information.

13. The mobile device of claim 12, comprising a sensor selected from the group consisting of a gyroscope, an accelerometer and a magnetometer.

14. The mobile device of claim 10, wherein the processor is configured to apply a Kalman filter or other estimator to compare the current image with reference image information.

15. The mobile device of claim 14, wherein the Kalman filter or other estimator is rewindable.

16. The mobile device of claim 10, wherein the processor is configured to determine the location and the orientation of the mobile device in real-time.

17. The mobile device of claim 10, wherein:

the processor is configured to add to the current image augmented reality information based on the determined location and orientation of the mobile device; and

the mobile device further comprises a display for showing the current image with the added augmented reality information.

18. A method of maintaining reference image information, comprising:

storing, in a server, reference image information in association with corresponding location information;

receiving, from a mobile device, an indication of a new, modified or missing reference image feature for a given location; and

incorporating, in the server, the new, modified or missing reference image feature corresponding to the given location.

19. The method of claim 18, comprising:

accumulating, in the server, a plurality of indications of the new, modified or missing reference image feature;

wherein updating the reference image feature corresponding to the given location follows reception of a predetermined number of indications.

20. The method of claim 19, wherein updating the reference image feature corresponding to the given location follows reception of the predetermined number of indications from a predetermined number of distinct mobile devices.

21. The method of claim 19, comprising:

recording times when the reference image features corresponding to the given location are updated;

receiving from a given mobile device, at the server, a request for reference image features for the given location;

selecting at the server, for the given location, reference image features having been stable for at least a predetermined period of time; and

sending toward the given mobile device, from the server, the selected reference image features.

22. A server, comprising:

a database for storing reference image information in association with corresponding location information;

a communication interface for receiving an indication of a new, modified or missing reference image feature for a given location; and

a processor for incorporating, in the database, the new, modified or missing reference image feature corresponding to the given location.

23. The server of claim 22, comprising:

a timer operably connected to the database for providing time values;

wherein;

the database is configured to record time values when reference image features corresponding to the given location are updated;

the interface is configured to receive a request for reference image features for the given location;

the processor is configured to select, for the given location, reference image features having been stable for at least a predetermined period of time; and

the interface is configured to send the selected reference image features.

24. The server of claim 22, wherein:

the database is configured to accumulate a plurality of indications of the new, modified or missing reference image feature;

wherein the processor is configured to update the reference image feature corresponding to the given location following accumulation of a predetermined number of indications in the database.