Late stage reprojection

• US 9,514,571 B2
• Filed: 07/25/2013
• Issued: 12/06/2016
• Est. Priority Date: 07/25/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Williams, Oliver Michael Christian, Barham, Paul, Isard, Michael, Wong, Tuan, Woo, Kevin, Klein, Georg, Service, Douglas Kevin, Michail, Ashraf Ayman, Pearson, Andrew, Shetter, Martin, Margolis, Jeffrey Neil, Ackerman, Nathan, Chan, Calvin, Tomlin, Arthur C.
• Primary Examiner(s): Gray, Ryan M
• Application Number:
  US13/951,351
• Publication Number:
  US 20150029218A1
• Time in Patent Office: 1,230 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 3/20 Linear translation of whole images or parts thereof, e.g. panning
• CPC Class Codes: G09G 2300/04 Structural and physical details of display devices
• CPC Class Codes: G09G 3/3208 organic, e.g. using organic light-emitting diodes [OLED]

Abstract

Methods for generating and displaying images associated with one or more virtual objects within an augmented reality environment at a frame rate that is greater than a rendering frame rate are described. The rendering frame rate may correspond with the minimum time to render images associated with a pose of a head-mounted display device (HMD). In some embodiments, the HMD may determine a predicted pose associated with a future position and orientation of the HMD, generate a pre-rendered image based on the predicted pose, determine an updated pose associated with the HMD subsequent to generating the pre-rendered image, generate an updated image based on the updated pose and the pre-rendered image, and display the updated image on the HMD. The updated image may be generated via a homographic transformation and/or a pixel offset adjustment of the pre-rendered image.

Claims

1. A mobile device, comprising:

one or more processors configured to generate a rendered image associated with a first predicted pose of the mobile device and determine a second predicted pose of the mobile device, the second predicted pose is different from the first predicted pose, the second predicted pose corresponds with a point in time during which a second updated image is displayed, the one or more processors configured to determine a pose difference between the first predicted pose and the second predicted pose and generate the second updated image based on the pose difference and at least a portion of the rendered image; and

a display in communication with the one or more processors, the display includes a pixel array, the display configured to store the second updated image within the pixel array and generate an updated image using pixel interconnections between pixels within the pixel array to shift pixel values within the pixel array, the display configured to display the updated image.

2. The mobile device of claim 1, wherein:

the pixel array includes a multiplexor configured to set a first pixel of the pixel array, the multiplexor configured to select one of a second voltage corresponding with a second pixel of the pixel array or a third voltage corresponding with a third pixel of the pixel array.

3. The mobile device of claim 1, wherein:

the one or more processors configured to identify a virtual object being focused on by an end user of the mobile device and determine a stabilization plane based on a location of the virtual object within an augmented reality environment, the one or more processors configured to generate the rendered image based on the stabilization plane.

4. The mobile device of claim 1, wherein:

the one or more processors configured to generate the second updated image by applying a homographic transformation to the at least a portion of the rendered image.

5. The mobile device of claim 4, wherein:

the homographic transformation comprises a multi-plane homography.

6. The mobile device of claim 4, wherein:

the homographic transformation comprises an affine homography.

7. The mobile device of claim 1, wherein:

the mobile device comprises an HMD; and

the display configured to generate the updated image based on an acceleration of the mobile device.

8. A method for displaying images, comprising:

generating a rendered image corresponding with a first predicted pose of a mobile device;

determining a second predicted pose of the mobile device, the second predicted pose is different from the first predicted pose and corresponds with a point in time during which at least a portion of an updated image is displayed using a display of the mobile device, the display includes a pixel array;

determining a pose difference between the first predicted pose and the second predicted pose;

generating the updated image based on the pose difference and at least a portion of the rendered image, the generating the updated image includes storing the rendered image within the pixel array and generating the updated image using pixel interconnections between pixels within the pixel array to shift pixel values within the pixel array; and

displaying the at least a portion of the updated image using the display of the mobile device.

9. The method of claim 8, wherein:

the second predicted pose corresponds with a middle display time for displaying the at least a portion of the updated image using the display; and

and

the pixel array includes a multiplexor configured to set a first pixel of the pixel array, the multiplexor configured to select one of a second voltage corresponding with a second pixel of the pixel array or a third voltage corresponding with a third pixel of the pixel array.

10. The method of claim 8, wherein:

the generating a rendered image includes identifying a virtual object being focused on by an end user of the mobile device, determining a stabilization plane based on a location of the virtual object within an augmented reality environment, and generating the rendered image based on the stabilization plane.

11. The method of claim 8, wherein:

the at least a portion of the updated image corresponds with one or more pixel rows within the updated image.

12. The method of claim 8, wherein:

the generating the updated image includes shifting the pixel values within the pixel array using the pixel interconnections between the pixels of the pixel array and a plurality of multiplexors embedded within the pixels of the pixel array.

13. The method of claim 8, wherein:

the mobile device comprises an HMD.

14. One or more storage devices containing processor readable code for programming one or more processors to perform a method for displaying images associated with one or more virtual objects using an HMD comprising the steps of:

determining a first predicted pose associated with the HMD;

generating a rendered image based on the first predicted pose;

determining a second predicted pose associated with the HMD subsequent to the determining a first predicted pose, the second predicted pose corresponds with a middle display time for displaying an updated image using a display of the HMD, the display includes a pixel array;

determining a pose difference between the first predicted pose and the second predicted pose;

generating the updated image based on the pose difference and at least a portion of the rendered image, the generating the updated image includes storing the rendered image within the pixel array and generating the updated image using pixel interconnections between pixels within the pixel array to shift pixel values within the pixel array; and

displaying the updated image using the display of the HMD.

15. The one or more storage devices of claim 14, wherein:

the generating a rendered image includes identifying a virtual object being focused on by an end user of the HMD, determining a stabilization plane based on a distance of the virtual object from the HMD, and generating the rendered image based on the stabilization plane.

16. The one or more storage devices of claim 14, wherein:

the pixel array includes a multiplexor configured to set a first pixel of the pixel array, the multiplexor configured to select one of a second voltage corresponding with a second pixel of the pixel array or a third voltage corresponding with a third pixel of the pixel array.

17. The one or more storage devices of claim 14, wherein:

the pixel array includes a plurality of multiplexors configured to set a plurality of pixel values within the pixel array.

18. The one or more storage devices of claim 14, wherein:

the generating the updated image includes applying a pixel offset adjustment to the at least a portion of the rendered image using the pixel interconnections between pixels within the pixel array.