Camera and sensor augmented reality techniques

• US 9,355,452 B2
• Filed: 12/19/2014
• Issued: 05/31/2016
• Est. Priority Date: 04/01/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Sugden, Benjamin J.
• Primary Examiner(s): DAGNEW, MEKONNEN D
• Application Number:
  US14/577,532
• Publication Number:
  US 20150103098A1
• Time in Patent Office: 529 Days
• Field of Search: 348/51, 348/207.1, 34833301-33306
• US Class Current: 1/1
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2210/04 Architectural design, interior design
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: H04N 5/2224 related to virtual studio applications

Abstract

Camera and sensor augmented reality techniques are described. In one or more implementations, an optical basis is obtained that was generated from data obtained by a camera of a computing device and a sensor basis is obtained that was generated from data obtained from one or more sensors that are not a camera. The optical basis and the sensor basis describe a likely orientation or position of the camera and the one or more sensors, respectively, in a physical environment. The optical basis and the sensor basis are compared to verify the orientation or the position of the computing device in the physical environment.

Claims

1. A method, comprising:

obtaining an optical basis generated from image data obtained by a camera of a computing device, the optical basis describing a likely orientation or position of the camera in a physical environment;

obtaining a sensor basis generated from sensor data obtained from one or more sensors of the computing device, the sensor basis describing a likely orientation or position of the one or more sensors in the physical environment; and

comparing the optical basis and the sensor basis to verify the orientation or the position of the computing device in the physical environment, the comparing comprising determining whether or not the orientation or the position described by the optical basis and the orientation or the position described by the sensor basis are consistent.

2. A method as described in claim 1, wherein the one or more sensors include an inertial measurement unit (IMU) that generates an IMU basis as at least part of the sensor basis, the IMU basis used to establish a higher degree of confidence to said verify the orientation or the position of the computing device in the physical environment.

3. A method as described in claim 1, wherein at least one of the one or more sensors is configured as a gyroscope, an accelerometer, or a magnetometer.

4. A method as described in claim 1, further comprising calculating a combined basis that indicates the likely orientation or position of the camera and the one or more sensors using the optical basis and the sensor basis.

5. A method as described in claim 1, further comprising generating an augmentation for display as part of an augmented reality display based at least in part on a current basis describing the likely orientation or position of the computing device in the physical environment.

6. A method as described in claim 1, further comprising, responsive to a determination that the optical basis and the sensor basis do not approximately correspond as a result of the comparing, extrapolating a current basis based at least in part on a previous basis of the computing device.

7. A method as described in claim 6, wherein the extrapolating is based at least in part on the previous basis and the obtained sensor basis.

8. A method as described in claim 7, wherein the previous basis is computed from a previous optical basis.

9. A method as described in claim 1, wherein the computing device includes a display device that is wearable by a user.

10. A method as described in claim 1, wherein determining whether the optical basis and the sensor basis are consistent comprises determining whether the optical basis and the sensor basis describe approximately a same orientation or position.

11. A method implemented by a computing device, the method comprising:

comparing an optical basis and a sensor basis to verify an orientation or position of the computing device in a physical environment, the comparing comprising determining whether a likely orientation or position of a camera in a physical environment described by the optical basis and a likely orientation or position of one or more sensors in the physical environment described by the sensor basis are consistent; and

responsive to a determination that the optical basis and the sensor basis are consistent as a result of the comparing, calculating a combined basis that describes the orientation or position of the computing device in the physical environment.

12. A method as described in claim 11, further comprising generating an augmentation for display as part of an augmented-reality display based at least in part on the combined basis, wherein the generated augmentation is configured as an object that appears to be added to a physical environment of the computing device.

13. A method as described in claim 12, further comprising displaying the augmentation by the computing device using one or more display devices.

14. A method as described in claim 11, wherein the one or more sensors of the computing device include a gyroscope, an accelerometer, or a magnetometer.

15. An apparatus, comprising:

a camera configured to capture image data;

an inertial measurement unit (IMU) configured to generate sensor data;

one or more modules that are configured to;

compute an optical basis from the image data and a sensor basis generated from the sensor data; and

compare the optical basis and the sensor basis to verify an orientation or position of the apparatus in a physical environment, the comparing comprising determining whether or not the orientation or the position described by the optical basis and the orientation or the position described by the sensor basis are consistent.

16. An apparatus as described in claim 15, wherein the inertial measure unit (IMU) includes a gyroscope, an accelerometer, and a magnetometer.

17. An apparatus as described in claim 15, further comprising:

a display device configured to be wearable by a user;

the one or more modules configured to;

calculate a combined basis that describes a likely orientation or position of the camera and the inertial measurement unit (IMU) in the physical environment; and

cause an augmented-reality display by the display device, the augmented-reality display including an augmentation generated based at least in part on the combined basis.

18. An apparatus as described in claim 17, wherein the likely orientation or position of the camera and the inertial measure unit (IMU) serve as a basis to determine an orientation or position of least one body part of a user.

19. An apparatus as described in claim 17, wherein the augmented-reality display includes the augmentation and an image captured using the camera.

20. An apparatus as described in claim 15, further comprising a housing that includes the one or more modules and that is configured as a tablet or a wireless phone.