Camera and sensor augmented reality techniques

• US 9,940,720 B2
• Filed: 05/18/2016
• Issued: 04/10/2018
• Est. Priority Date: 04/01/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Sugden, Benjamin J.
• Primary Examiner(s): DAGNEW, MEKONNEN D
• Application Number:
  US15/157,964
• Publication Number:
  US 20160267662A1
• Time in Patent Office: 692 Days
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2210/04 Architectural design, interior design
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: H04N 5/2224 related to virtual studio applications

Abstract

Camera and sensor augmented reality techniques are described. In one or more implementations, sensor data is obtained from a sensor of a hardware device, the sensor data being associated with the hardware device that is located in an environment, such as in three-dimensional (3D) space. Images of the environment are captured with at least one camera of the hardware device. A position of the hardware device in the environment can then be determined based on at least one of the sensor data and the images of the environment. Further, an orientation of the hardware device in the environment can be determined based on at least one of the sensor data and the images of the environment.

Claims

1. A method to determine a position of a hardware device in an environment, the method comprising:

obtaining sensor data from a sensor of the hardware device, the sensor data associated with the hardware device that is located in the environment;

capturing images of the environment viewable with at least one camera of the hardware device;

determining the position of the hardware device in the environment indicated by the images of the environment; and

verifying that the position of the hardware device as determined from the images is consistent with an inertial measurement based on the sensor data.

2. The method as recited in claim 1, wherein the environment is in three dimensional (3D) space, and said determining the position of the hardware device in the 3D space.

3. The method as recited in claim 1, wherein the hardware device is a user device comprising the sensor, the at least one camera, and an augmented reality module stored as software instructions executable by a processor to perform said determining the position of the hardware device in the environment.

4. The method as recited in claim 1, wherein the sensor of the hardware device is an inertial measurement unit (IMU), and said obtaining the sensor data as IMU data from the inertial measurement unit.

5. The method as recited in claim 1, further comprising:

determining an orientation of the hardware device in the environment based on at least one of the sensor data and the images of the environment.

6. The method as recited in claim 1, further comprising:

determining a sensor basis of the position of the hardware device based on the sensor data; and

said determining the position of the hardware device in the environment based on the sensor basis.

7. The method as recited in claim 1, further comprising:

determining an optical basis of the position of the hardware device based on the images of the environment; and

said determining the position of the hardware device in the environment based on the optical basis.

8. The method as recited in claim 1, further comprising:

determining a sensor basis of the position of the hardware device based on the sensor data;

determining an optical basis of the position of the hardware device based on the images of the environment; and

said determining the position of the hardware device in the environment based on the sensor basis being consistent with the optical basis as to the position of the hardware device.

9. A hardware device, comprising:

an inertial measurement unit configured to generate sensor data associated with the hardware device located in an environment;

at least one camera configured to capture images of the environment;

a memory configured to store an augmented reality module as software instructions executable by a processor system of the hardware device, the augmented reality module implemented to;

receive the sensor data from the inertial measurement unit;

receive the images of the environment from the at least one camera;

determine a position of the hardware device in the environment indicated by the images of the environment and

verify that the position of the hardware device as determined from the images is consistent with an inertial measurement based on the sensor data.

10. The hardware device as recited in claim 9, wherein the environment is in three dimensional (3D) space, and the augmented reality module is implemented to determine the position of the hardware device in the 3D space.

11. The hardware device as recited in claim 9, wherein the augmented reality module is implemented to:

determine a sensor basis of the position of the hardware device based on the sensor data; and

determine the position of the hardware device in the environment based on the sensor basis.

12. The hardware device as recited in claim 9, wherein the augmented reality module is implemented to:

determine an optical basis of the position of the hardware device based on the images of the environment; and

determine the position of the hardware device in the environment based on the optical basis.

13. The hardware device as recited in claim 9, wherein the augmented reality module is implemented to:

determine a sensor basis of the position of the hardware device based on the sensor data;

determine an optical basis of the position of the hardware device based on the images of the environment; and

determine the position of the hardware device in the environment based on the sensor basis being consistent with the optical basis as to the position of the hardware device.

14. The hardware device as recited in claim 9, wherein the augmented reality module is implemented to determine an orientation of the hardware device in the environment based on at least one of the sensor data and the images of the environment.

15. A method to determine a spatial-positioning of camera-based hardware in three-dimensional (3D) space, the method comprising:

obtaining sensor data from a sensor of the camera-based hardware;

capturing images of the 3D space with at least one camera of the camera based hardware;

determining the spatial-positioning of the camera-based hardware in the 3D space indicated by the images of the 3D space, the images including markers in the 3D space that are recognized and serve as a basis to determine the spatial positioning of the camera-based hardware; and

verifying that the spatial-positioning of the camera-based hardware as determined from the images of the 3D space is consistent with an inertial measurement based on the sensor data.

16. The method as recited in claim 15, further comprising:

determining an orientation of the camera-based hardware in the 3D space based on the images of the 3D space.

17. The method as recited in claim 16, further comprising:

determining the orientation of the camera-based hardware in the 3D space based on the sensor data.

18. The method as recited in claim 15, further comprising:

determining a sensor basis of the spatial-positioning of the camera-based hardware based on the sensor data; and

said determining the spatial-positioning of the camera-based hardware in the 3D space based on the sensor basis.

19. The method as recited in claim 15, further comprising:

determining an optical basis of the spatial-positioning of the camera-based hardware based on the images of the 3D space; and

said determining the spatial-positioning of the camera-based hardware in the 3D space based on the optical basis.

20. The method as recited in claim 15, further comprising:

verifying that the spatial-positioning of the camera-based hardware as determined from the images of the 3D space is consistent with a verified spatial positioning of the camera-based hardware from a previous point in time.