Late stage reprojection

• US 9,747,726 B2
• Filed: 08/03/2016
• Issued: 08/29/2017
• Est. Priority Date: 07/25/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Williams, Oliver Michael Christian, Barham, Paul, Isard, Michael, Wong, Tuan, Woo, Kevin, Klein, Georg, Service, Douglas Kevin, Michail, Ashraf Ayman, Pearson, Andrew, Shetter, Martin, Margolis, Jeffrey Neil, Ackerman, Nathan, Chan, Calvin, Tomlin, Arthur C.
• Primary Examiner(s): Gray, Ryan M
• Application Number:
  US15/227,453
• Publication Number:
  US 20160343172A1
• Time in Patent Office: 391 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 3/20 Linear translation of whole images or parts thereof, e.g. panning
• CPC Class Codes: G09G 2300/04 Structural and physical details of display devices
• CPC Class Codes: G09G 3/3208 organic, e.g. using organic light-emitting diodes [OLED]

Abstract

Methods for generating and displaying images associated with one or more virtual objects within an augmented reality environment at a frame rate that is greater than a rendering frame rate are described. The rendering frame rate may correspond with the minimum time to render images associated with a pose of a head-mounted display device (HMD). In some embodiments, the HMD may determine a predicted pose associated with a future position and orientation of the HMD, generate a pre-rendered image based on the predicted pose, determine an updated pose associated with the HMD subsequent to generating the pre-rendered image, generate an updated image based on the updated pose and the pre-rendered image, and display the updated image on the HMD. The updated image may be generated via a homographic transformation and/or a pixel offset adjustment of the pre-rendered image.

Claims

1. An electronic device for providing an augmented reality environment, comprising:

a control circuit configured to acquire a rendered image associated with a first predicted pose of the electronic device and identify a first homographic transformation to be applied to a first portion of the rendered image based on a difference between the first predicted pose and a second predicted pose of the electronic device, the control circuit configured to generate a first portion of an updated image by applying the first homographic transformation to the first portion of the rendered image; and

a see-through display configured to display a second portion of the rendered image and the first portion of the updated image at the same time, the see-through display comprises an integrated circuit with a pixel array and the control circuit, the see-through display configured to store the first portion of the rendered image within the pixel array and generate the first portion of the updated image using pixel interconnections between pixels within the pixel array to shift pixel values within the pixel array.

2. The electronic device of claim 1, wherein:

the first portion of the rendered image is associated with a first color field of the rendered image and the second portion of the rendered image is associated with a second color field of the rendered image.

3. The electronic device of claim 1, wherein:

the first predicted pose corresponds with a first point in time and the second predicted pose corresponds with a second point in time subsequent to the first point in time.

4. The electronic device of claim 1, wherein:

the electronic device comprises a head-mounted display device.

5. A method for providing an augmented reality environment, comprising:

acquiring a rendered image at a mobile device, the rendered image is associated with a first predicted pose of the mobile device;

determining a first homographic transformation to be applied to a first portion of the rendered image based on a difference between the first predicted pose of the mobile device and a second predicted pose of the mobile device;

generating a first portion of an updated image by applying the first homographic transformation to the first portion of the rendered image, the generating the first portion of the updated image is performed by one or more control circuits integrated within a display of the mobile device, the display comprises an integrated circuit with a pixel array, the generating the first portion of the updated image includes storing the first portion of the rendered image within the pixel array and generating the first portion of the updated image using pixel interconnections between pixels within the pixel array to shift pixel values within the pixel array; and

concurrently displaying a second portion of the rendered image and the first portion of the updated image using the mobile device.

6. The method of claim 5, wherein:

the first portion of the rendered image is associated with a first color field of the rendered image and the second portion of the rendered image is associated with a second color field of the rendered image.

7. The method of claim 5, wherein:

the display comprises an OLED display; and

the mobile device comprises a head-mounted display device.

8. The method of claim 5, wherein:

the first homographic transformation comprises an affine transformation.

9. The method of claim 5, wherein:

the applying the first homographic transformation to the first portion of the rendered image includes applying a weighted mapping to a plurality of pixels within the first portion of the rendered image.

10. The method of claim 5, wherein:

the generating the first portion of the updated image includes applying a first pixel offset adjustment to the first portion of the rendered image.

11. The method of claim 10, wherein:

the applying the first pixel offset adjustment includes loading a first image associated with the first portion of the rendered image into the pixel array of the display and shifting the first image while the first image is stored within the pixel array using circuitry integrated within the pixel array.

12. The method of claim 10, wherein:

the applying the first pixel offset adjustment includes loading a first image associated with the first portion of the rendered image into a buffer of the display and shifting the first image within the buffer.

13. A method for providing an augmented reality environment, comprising:

acquiring a rendered image at a mobile device, the rendered image includes a first portion of the rendered image associated with a first color field and a second portion of the rendered image associated with a second color field;

generating a first portion of an updated image by applying a first homographic transformation to the first portion of the rendered image, the first homographic transformation is associated with a difference between a first pose of the mobile device and a second pose of the mobile device, the mobile device includes a display, the display includes a pixel array, the generating the first portion of the updated image includes storing the first portion of the rendered image within the pixel array and generating the first portion of the updated image using pixel interconnections between pixels within the pixel array to shift pixel values within the pixel array;

generating a second portion of the updated image by applying a second homographic transformation to the second portion of the rendered image, the second homographic transformation is associated with a difference between the first pose of the mobile device and a third pose of the mobile device; and

displaying the first portion of the updated image and the second portion of the updated image using the mobile device.

14. The method of claim 13, wherein:

the displaying the first portion of the updated image and the second portion of the updated image comprises displaying the first portion of the updated image and the second portion of the updated image at the same time.

15. The method of claim 13, wherein:

the mobile device comprises a head-mounted display device.