Augmented reality computing with inertial sensors

• US 9,024,972 B1
• Filed: 04/01/2009
• Issued: 05/05/2015
• Est. Priority Date: 04/01/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Bronder, Matthew L., Dougherty, Michael A., Green, Adam, Bertolami, Joseph, Craig, Robert M.
• Primary Examiner(s): Brier, Jeffrey A
• Application Number:
  US12/416,321
• Time in Patent Office: 2,225 Days
• Field of Search: 345/633
• US Class Current: 345/633
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 3/60 Rotation of whole images or parts thereof

Abstract

Example embodiments of the present disclosure provide techniques for receiving measurements from one or more inertial sensors (i.e. accelerometer and angular rate gyros) attached to a device with a camera or other environment capture capability. In one embodiment, the inertial measurements may be combined with pose estimates obtained from computer vision algorithms executing with real time camera images. Using such inertial measurements, a system may more quickly and efficiently obtain higher accuracy orientation estimates of the device with respect to an object known to be stationary in the environment.

Claims

1. In a computing device comprising a processor, memory, and a display, a method for augmenting location based data, the method comprising:

receiving, from an image capture device, image data representative of at least one object in a vicinity of a first location and a first orientation for said image capture device;

determining the first location and first orientation as a function of at least one geographically invariant point determined from said image data, the geographically invariant point determined based on one or more feature descriptors that are extracted from an object in an image that can be used to distinguish the object from other objects in the image;

determining a second location and a second orientation of said image capture device, the second location and second orientation determined as a function of inertial sensor information indicative of a relative movement of said image capture device and said first location and said first orientation;

sending, to a server via a communications network, the second location and second orientation;

receiving, from the server via the communications network, a media entity associated with said second location and said second orientation and metadata indicative of a type of data associated with the media entity and information for rendering the media entity on the display, said media entity determined based at least on the second location and second orientation and usable to augment images of the second location captured by said image capture device; and

rendering the media entity on said display based on said second location and second orientation.

2. The method of claim 1, wherein said inertial sensor information comprises at least one of accelerometer data or angular velocity data.

3. The method of claim 1, wherein said determining comprises discarding said first location and said first orientation when said first location and said first orientation meets a predetermined criterion.

4. The method of claim 2, wherein said accelerometer data is received from triaxial accelerometers mounted on said image capture device such that each accelerometer axis deviates from a gravity vector when said image capture device is oriented for typical device usage.

5. The method of claim 1, wherein said inertial sensor information comprises data sampled at a rate that at least matches a capture rate of said image capture device.

6. The method of claim 1, wherein said second location and second orientation are determined by combining said inertial sensor information with said first location and first orientation.

7. The method of claim 1, wherein said determining the second location and the second orientation comprises applying an estimation filter.

8. The method of claim 1, wherein a measurement bias estimate is adjusted as a function of said second location, said second orientation, and said inertial sensor information.

9. The method of claim 1, further comprising:

storing said second location and second orientation;

receiving from the image capture device updated image data;

receiving updated inertial sensor information; and

determining a third location and a third orientation of said image capture device, the third location and third orientation determined as a function of said updated inertial sensor information and said second location and said second orientation.

10. The method of claim 9, further comprising storing said updated inertial sensor information when the updated inertial sensor information is received while the third location and third orientation are being determined.

11. The method of claim 10, further comprising updating the stored second location and second orientation as a function of the updated inertial sensor information and correcting the updated inertial sensor information as a function of a sensor bias.

12. The method of claim 10, further comprising:

when said third location and third orientation has been determined, updating the third location and third orientation as a function of the updated inertial sensor information.

13. The method of claim 11, wherein updating the stored second location and second orientation is performed as a function of the at least one geographically invariant point and the received inertial sensor information.

14. The method of claim 13, further comprising narrowing a field of search for the at least one geographically invariant point by predicting a new location of the at least one geographically invariant point based on the received inertial sensor information.

15. The method of claim 14, further comprising eliminating search zones that conflict with the predicted new location.

16. A system configured to augment location based data, comprising:

at least one processor;

a data store; and

at least one memory communicatively coupled to said at least one processor, the memory having stored therein computer-executable instructions that, when executed, cause the system to performs steps comprising;

receive from an image capture device at least one set of image data representative of at least one object;

determine a first location and first orientation for said image capture device as a function of at least one geographically invariant point on said at least one object, the geographically invariant point determined based on one or more feature descriptors that are extracted from an object to distinguish the object from other objects in an image;

determine a second location and a second orientation of said image capture device, the second location and second orientation determined as a function of inertial sensor information indicative of a relative movement of said image capture device and said first location and said first orientation;

send, to a server via a communications network, the at least one set of image data and the second location and second orientation;

receive, from the server via the communications network, a media entity associated with said second location and said second orientation and metadata indicative of a type of data associated with the media entity and information for rendering the media entity on a display device, said media entity determined based at least on the second location and second orientation and usable to augment images of the second location captured by said image capture device; and

render the media entity on the display device based on said second location and second orientation.

17. The system of claim 16, wherein said determine the second location and the second orientation comprises applying an estimation filter.

18. The system of claim 16, further comprising estimating the relative movement of said image capture device based on the inertial sensor information.

19. The system of claim 18, wherein said determine the second location and the second orientation further comprises searching for the at least one geographically invariant point in accordance with the estimated relative movement of said image capture device.

20. A computing device comprising:

at least one processor; and

at least one memory communicatively coupled to said at least one processor, the memory having stored therein computer-executable instructions that, when executed by the at least one processor, cause the computing device to;

receive from an image capture device at least one set of image data representative of at least one object;

receive inertial sensor information indicating a relative movement of said image capture device and estimating a first location and first orientation for said image capture device using the inertial sensor information;

search for the at least one geographically invariant point in accordance with the estimated first location and first orientation and determining the first location and first orientation for said image capture device as a function of the at least one geographically invariant point, the geographically invariant point determined based on one or more feature descriptors that are extracted from an object in an image and comprise a point or part of interest in the object;

determine a second location and a second orientation of said image capture device, the second location and second orientation determined by combining said inertial sensor information indicating the relative movement of said image capture device and said first location and said first orientation;

send, to a server via a communications network, the at least one set of image data and the second location and second orientation;

receive, from the server via the communications network, a media entity associated with said second location and said second orientation and metadata indicative of a type of data associated with the media entity and information for rendering the media entity on a display, said media entity determined based at least on the second location and second orientation and usable to augment images of the second location captured by said image capture device; and

render the media entity on said display based on said second location and second orientation.