Augmented reality extrapolation techniques

• US 9,613,463 B2
• Filed: 01/08/2016
• Issued: 04/04/2017
• Est. Priority Date: 04/20/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Sugden, Benjamin J.
• Primary Examiner(s): Du, Haixia
• Application Number:
  US14/991,729
• Publication Number:
  US 20160125658A1
• Time in Patent Office: 452 Days
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2320/02 Improving the quality of display appearance
• CPC Class Codes: G09G 2320/0252 Improving the response speed
• CPC Class Codes: G09G 2320/0261 in the context of movement of objects on the screen or movement of the observer relative to the screen
• CPC Class Codes: G09G 2320/08 Arrangements within a display terminal for setting, manually or automatically, display parameters of the display terminal
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 2360/12 Frame memory handling
• CPC Class Codes: G09G 2380/00 Specific applications
• CPC Class Codes: G09G 3/003 to produce spatial visual effects

Abstract

Augmented reality extrapolation techniques are described. In one or more implementations, a frame of an augmented-reality display is rendered based at least in part on an optical basis that describes a current orientation or position of at least a part of a computing device. While the frame is rendered, an extrapolation based on a previous basis and a sensor basis generates an updated optical basis that describes a likely orientation or position of the part of the computing device, and the extrapolation is effective to account for a lag time duration between rendering the frame and displaying the frame of the augmented-reality display. The rendered frame of the augmented-reality display is updated before the rendered frame is displayed based at least in part on the updated optical basis that describes the likely orientation or position of the part of the computing device.

Claims

1. A method implemented by a computing device, the method comprising:

rendering a frame of an augmented-reality display based at least in part on an optical basis that describes a current orientation or position of at least a part of the computing device;

receiving, while rendering the frame, sensor data from one or more sensors;

generating, during the rendering of the frame, a sensor basis based on the sensor data obtained from an inertial measurement unit of the computing device that describes the current orientation or position of at least the part of the computing device;

extrapolating, while rendering the frame, based on a previous basis and the sensor basis to generate an updated optical basis that describes a likely orientation or position of the part of the computing device, the previous basis being a previous optical basis or a previous sensor basis prior to the optical basis that describes the current orientation or position, the sensor basis based on the sensor data received while rendering the frame, said extrapolating effective to account for a lag time duration between said rendering the frame and displaying the frame of the augmented-reality display; and

updating the rendered frame of the augmented-reality display before the rendered frame is displayed, said updating based at least in part on the updated optical basis that describes the likely orientation or position of the part of the computing device.

2. A method as described in claim 1, wherein:

the rendering the frame of the augmented-reality display causes the frame to be stored in a frame buffer of the computing device; and

the updating is performed on the rendered frame of the augmented-reality display that is stored in the frame buffer of the computing device.

3. A method as described in claim 1, wherein the updating is performed before said displaying the rendered augmented-reality display by a display device of the computing device.

4. A method as described in claim 1, further comprising:

determining, during the rendering of the frame, that the optical basis is not consistent with the sensor basis to describe the likely orientation or position of the part of the computing device.

5. A method as described in claim 4, wherein the inertial measurement unit includes at least one of a gyroscope, an accelerometer, or a magnetometer.

6. A method as described in claim 1, wherein the extrapolating uses the optical basis and at least one of the sensor data and optical data obtained subsequent to calculation of the optical basis to produce an extrapolated second basis.

7. A method as described in claim 1, wherein the optical basis is generated based on camera optical data and the extrapolating generates the updated optical basis by integrating velocities read from the one or more sensors of the computing device that are not a camera and compositing the velocities with the optical basis that was used in the rendering.

8. A method as described in claim 1, wherein the augmented-reality display includes a view of an image captured by a camera of the computing device and an object generated by the computing device to be added to the image, the generation of the object based at least in part on the optical basis.

9. A method implemented by a computing device, the method comprising:

calculating a first basis from a sensor that describes a likely position or orientation of at least a part of the computing device;

generating an augmentation by the computing device for display as part of an augmented-reality display based at least in part on the first basis;

calculating a second basis from another, different sensor that describes an updated orientation or position to which the part of the computing device was moved during the rendering of the augmented-reality display;

determining, during the rendering of the augmented-reality display, that the first basis is not consistent with the second basis to describe the likely position or orientation of the part of the computing device;

calculating, while rendering the augmented-reality display, a modification to be applied to the rendered augmented-reality display that includes the augmentation, the modification based on the second basis; and

updating the rendered augmented-reality display before the rendering is displayed, said updating the rendered augmented-reality display based on the calculated modification.

10. A method as described in claim 9, wherein the generated augmentation is effective to implement an object appearing to be added to a physical environment of the computing device.

11. A method as described in claim 9, wherein the second basis is computed by interpolating the first basis using data obtained from one or more sensors of the computing device, the one or more sensors configured as an inertial measurement unit that includes at least one of a gyroscope, an accelerometer, or a magnetometer.

12. A method as described in claim 9, further comprising:

displaying the augmented-reality display by the computing device using one or more display devices.

13. A method as described in claim 9, wherein the first basis is formed by verifying an optical basis calculated using optical data taken by a camera of the computing device with a sensor basis calculated using sensor data obtained from one or more sensors that are not a camera.

14. An apparatus comprising:

a frame buffer;

one or more sensors that describe a likely orientation or position of at least a part of the apparatus in a physical environment; and

at least one processor and a memory configured to store one or more modules executable by the at least one processor and configured to;

calculate a first basis from a sensor of the one or more sensors that describes the likely orientation or position of the part of the apparatus in the physical environment;

render a frame of an augmented-reality display to the frame buffer that is based at least in part on the first basis;

obtain, while the frame is being rendered, sensor data from a different sensor of the one or more sensors, the different sensor describing an updated orientation or position of the part of the apparatus in the physical environment while the frame is rendered in the frame buffer;

calculate, while the frame is being rendered, a second basis using the first basis and the obtained sensor data from the different sensor;

extrapolate, while the frame is being rendered, an updated first basis using the second basis, the extrapolation effective to account for a lag time duration between the frame being rendered to the frame buffer and the frame being displayed; and

update the frame in the frame buffer before the frame is displayed, the frame updated based on the extrapolated updated first basis.

15. An apparatus as described in claim 14, wherein the one or more modules are further configured to:

determine, while the frame is being rendered, that the first basis does not correspond with the second basis.

16. An apparatus as described in claim 14, wherein the one or more sensors include at least one of an inertial measure unit which includes a gyroscope, an accelerometer, or a magnetometer.

17. An apparatus as described in claim 14, further comprising a display device to display the rendered frame, the display device configured to be wearable by a user.

18. An apparatus as described in claim 17, wherein the rendered frame is updated in the frame buffer before the frame is displayed by the display device.

19. An apparatus as described in claim 14, wherein the one or more sensors are not a camera and the first basis is formed at least in part using optical data obtained via a camera.

20. An apparatus as described in claim 14, wherein the augmented-reality display includes an augmentation and an image captured using a camera of the apparatus.