Augmented reality computing with inertial sensors

• US 9,761,054 B2
• Filed: 05/04/2015
• Issued: 09/12/2017
• Est. Priority Date: 04/01/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Bronder, Matthew L., Dougherty, Michael A., Green, Adam, Bertolami, Joseph, Craig, Robert M.
• Primary Examiner(s): Brier, Jeffery A
• Application Number:
  US14/703,634
• Publication Number:
  US 20150235432A1
• Time in Patent Office: 862 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 3/60 Rotation of whole images or parts thereof

Abstract

Example embodiments of the present disclosure provide techniques for receiving measurements from one or more inertial sensors (i.e. accelerometer and angular rate gyros) attached to a device with a camera or other environment capture capability. In one embodiment, the inertial measurements may be combined with pose estimates obtained from computer vision algorithms executing with real time camera images. Using such inertial measurements, a system may more quickly and efficiently obtain higher accuracy orientation estimates of the device with respect to an object known to be stationary in the environment.

Claims

1. In a computing device comprising a processor, memory, and a display, a method for augmenting location based data, the method comprising:

receiving, from an image capture device, a location determination and image data representative of at least one object;

determining a first location and first orientation as a function of the location determination and at least one geographically invariant point determined from said image data, the geographically invariant point determined based on one or more feature descriptors that are extracted from an object in an image that can be used to distinguish the object from other objects in the image;

sending, to a server via a communications network, the first location and first orientation;

receiving, from the server via the communications network, a media entity associated with said first location and said first orientation and metadata indicative of a type of data associated with the media entity and information for rendering the media entity on the display, said media entity determined based at least on the first location and first orientation and usable to augment images of the first location captured by said image capture device; and

rendering the media entity on said display based on said first location and first orientation.

2. The method of claim 1, wherein said location determination is determined using a location determination method performed by a device coupled to the image capture device.

3. The method of claim 1, further comprising determining a second location and a second orientation of said image capture device, the second location and second orientation determined as a function of inertial sensor information indicative of a relative movement of the image capture device and said first location and said first orientation.

4. The method of claim 3, wherein said inertial sensor information comprises at least one of accelerometer data or angular velocity data.

5. The method of claim 4, wherein said accelerometer data is received from triaxial accelerometers mounted on said image capture device such that each accelerometer axis deviates from a gravity vector when said image capture device is oriented for typical device usage.

6. The method of claim 3, wherein said inertial sensor information comprises data sampled at a rate that at least matches a capture rate of said image capture device.

7. The method of claim 3, wherein said second location and second orientation are determined by combining said inertial sensor information with said first location and first orientation.

8. The method of claim 3, wherein said determining the second location and the second orientation comprises applying an estimation filter.

9. The method of claim 3, wherein a measurement bias estimate is adjusted as a function of said second location, said second orientation, and said inertial sensor information.

10. The method of claim 3, further comprising:

storing said second location and second orientation;

receiving updated image data from the image capture device;

receiving updated inertial sensor information; and

determining a third location and a third orientation of said image capture device, the third location and third orientation determined as a function of said updated inertial sensor information and said second location and said second orientation.

11. The method of claim 1, wherein said media entity comprises at least one of an audio file, image file, text file, animation file, geometry data, or cartography data.

12. The method of claim 1, further comprising receiving inputs from a user for generating a user defined media entity.

13. A system configured to augment location based data, comprising:

at least one processor;

a data store; and

at least one memory communicatively coupled to said at least one processor, the memory having stored therein computer-executable instructions that, when executed, cause the system to perform steps comprising;

receiving from an image capture device a location determination and at least one set of image data representative of at least one object;

determining a first location and first orientation for said image capture device as a function of the location determination and at least one geographically invariant point on said at least one object, the geographically invariant point determined based on one or more feature descriptors that are extracted from an object to distinguish the object from other objects in an image;

sending, to a server via a communications network, the at least one set of image data and the first location and first orientation;

receiving, from the server via the communications network, a media entity associated with said first location and said first orientation and metadata indicative of a type of data associated with the media entity and information for rendering the media entity on a display device, said media entity determined based at least on the first location and first orientation and usable to augment images of the first location captured by said image capture device; and

rendering the media entity on the display device based on said first location and first orientation.

14. The system of claim 13, further comprising computer-executable instructions that, when executed, cause the system to performs steps comprising:

determining a second location and a second orientation of said image capture device, the second location and second orientation determined as a function of inertial sensor information indicative of a relative movement of said image capture device and said first location and said first orientation.

15. The system of claim 14, wherein said determine the second location and the second orientation comprises applying an estimation filter.

16. The system of claim 15, wherein the relative movement of said image capture device is estimated based on the inertial sensor information.

17. The system of claim 16, wherein the second location and the second orientation are determined in part by searching for the at least one geographically invariant point in accordance with the estimated relative movement of said image capture device.

18. A computing device comprising:

at least one processor; and

at least one memory communicatively coupled to said at least one processor, the memory having stored therein computer-executable instructions that, when executed by the at least one processor, cause the computing device to;

receive from an image capture device a location determination and at least one set of image data representative of at least one object;

estimate a first location and first orientation for said image capture device based on the location determination;

search for at least one geographically invariant point in accordance with the estimated first location and first orientation and determine an updated first location and first orientation for said image capture device as a function of the at least one geographically invariant point, the geographically invariant point determined based on one or more feature descriptors that are extracted from an object in an image and comprise a point or part of interest in the object;

send, to a server via a communications network, the at least one set of image data and the updated first location and first orientation;

receive, from the server via the communications network, a media entity associated with said updated first location and first orientation and metadata indicative of a type of data associated with the media entity and information for rendering the media entity on a display, said media entity determined based at least on the updated first location and first orientation and usable to augment images of the updated first location captured by said image capture device; and

render the media entity on said display based on said updated first location and first orientation.

19. The computing device of claim 18, further computer-executable instructions that, when executed by the at least one processor, cause the computing device to:

receive inertial sensor information indicating a relative movement of said image capture device; and

estimate a second location and second orientation for said image capture device using the inertial sensor information.

20. The computing device of claim 19, wherein said second location and a second orientation is determined by combining said inertial sensor information indicating the relative movement of said image capture device and said updated first location and said first orientation.