Camera and sensor augmented reality techniques

• US 8,937,663 B2
• Filed: 04/01/2011
• Issued: 01/20/2015
• Est. Priority Date: 04/01/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Sugden, Benjamin J.
• Primary Examiner(s): DAGNEW, MEKONNEN D
• Application Number:
  US13/078,425
• Publication Number:
  US 20120249807A1
• Time in Patent Office: 1,390 Days
• Field of Search: 348/51, 348/207.1, 34833301-33306
• US Class Current: 348/207.1
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2210/04 Architectural design, interior design
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: H04N 5/2224 related to virtual studio applications

Abstract

Camera and sensor augmented reality techniques are described. In one or more implementations, an optical basis is obtained that was generated from data obtained by a camera of a computing device and a sensor basis is obtained that was generated from data obtained from one or more sensors that are not a camera. The optical basis and the sensor basis describe a likely orientation or position of the camera and the one or more sensors, respectively, in a physical environment. The optical basis and the sensor basis are compared to verify the orientation or the position of the computing device in the physical environment.

Claims

1. A method implemented by a computing device, the method comprising:

obtaining an optical basis generated from image data obtained by a camera of the computing device, the optical basis describing a likely orientation or position of the camera in a physical environment;

obtaining a sensor basis generated from sensor data obtained from one or more sensors of the computing device, the sensor basis describing a likely orientation or position of the one or more sensors in the physical environment, the one or more sensors including an inertial measurement unit (IMU) that generates an IMU basis as at least part of the sensor basis;

comparing the optical basis and the sensor basis to verify the orientation or the position of the computing device in the physical environment; and

responsive to a determination that the optical basis and the sensor basis do not approximately correspond as a result of the comparing, extrapolating a current basis based at least in part on a previous basis of the computing device, the current basis describing a likely orientation or position of the computing device in the physical environment.

2. A method as described in claim 1, wherein the one or more sensors include the inertial measurement unit (IMU) that generates the IMU basis used to establish a higher degree of confidence to said verify the orientation or the position of the computing device in the physical environment.

3. A method as described in claim 1, wherein at least one of the one or more sensors is configured as a gyroscope, an accelerometer, or a magnetometer.

4. A method as described in claim 1, further comprising generating an augmentation for display as part of an augmented reality display based at least in part on the current basis.

5. A method as described in claim 1, wherein the extrapolating is based at least in part on the previous basis and the obtained sensor basis.

6. A method as described in claim 5, wherein the previous basis is computed from a previous optical basis.

7. A method as described in claim 1, wherein the computing device is configured as a wireless telephone or a tablet.

8. A method as described in claim 1, wherein the computing device includes a display device that is wearable by a user.

9. A method implemented by a computing device, the method comprising:

comparing an optical basis and a sensor basis to verify an orientation or position of the computing device, the optical basis computed from image data received by a camera of the computing device and describing a likely orientation or position of the camera in a physical environment, the sensor basis generated from sensor data obtained from one or more sensors of the computing device that are not the camera and describing a likely position or orientation of the one or more sensors in the physical environment;

responsive to a determination that the optical basis and the sensor basis approximately correspond as a result of the comparing, calculating a combined basis that describes the orientation or position of the computing device in the physical environment; and

generating an augmentation by the computing device for display as part of an augmented-reality display based at least in part on the combined basis.

10. A method as described in claim 9, wherein the generated augmentation is configured such that an object appears to be added to a physical environment of the computing device.

11. A method as described in claim 9, wherein the one or more sensors of the computing device include a gyroscope, an accelerometer, or a magnetometer.

12. A method as described in claim 9, further comprising displaying the augmentation by the computing device using one or more display devices.

13. A method as described in claim 9, wherein the one or more sensors include an inertial measurement unit (IMU) that generates an IMU basis as at least part of the sensor basis.

14. A method as described in claim 13, wherein the IMU basis is used to establish a higher degree of confidence to verify the orientation or the position of the computing device in the physical environment.

15. An apparatus comprising:

a camera;

an inertial measurement unit (IMU);

a display device; and

one or more modules that are configured to;

compute an optical basis from image data obtained by the camera, the optical basis describing a likely orientation or position of the camera in a physical environment;

compute a sensor basis generated from sensor data obtained from the inertial measurement unit (IMU), the sensor basis describing a likely orientation or position of the IMU in the physical environment;

compare the optical basis to the sensor basis to verify the likely orientation or position of the camera and the IMU in the physical environment;

responsive to a determination that the optical basis and the sensor basis approximately correspond as a result of the comparing, calculate a combined basis that describes a likely orientation or position of the camera and the inertial measurement unit (IMU) in the physical environment; and

cause an augmented-reality display to be displayed by the display device, the augmented-reality display including an augmentation that was generated based at least in part on the combined basis.

16. An apparatus as described in claim 15, wherein the inertial measure unit (IMU) includes a gyroscope, an accelerometer, and a magnetometer.

17. An apparatus as described in claim 15, wherein the display device is configured to be wearable by a user.

18. An apparatus as described in claim 15, wherein the likely orientation or position of the camera and the inertial measure unit (IMU) serve as a basis to determine a like orientation or position of least one body part of a user.

19. An apparatus as described in claim 15, further comprising a housing that includes the one or more modules and that is configured as a tablet or a wireless phone.

20. An apparatus as described in claim 15, wherein the augmented-reality display includes the augmentation and an image captured using the camera.