Reprojection OLED display for augmented reality experiences

• US 9,892,565 B2
• Filed: 07/28/2017
• Issued: 02/13/2018
• Est. Priority Date: 06/28/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Chan, Calvin, Margolis, Jeffrey Neil, Pearson, Andrew, Shetter, Martin, Michail, Ashraf Ayman, Corlett, Barry
• Primary Examiner(s): ZALALEE, SULTANA MARCIA
• Application Number:
  US15/663,521
• Publication Number:
  US 20170345217A1
• Time in Patent Office: 200 Days
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 3/02 Affine transformations for image registration G06T3/147; for image mosaicing G06T3/4038
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G09G 2300/0804 Sub-multiplexed active matrix panel, i.e. wherein one active driving circuit is used at pixel level for multiple image producing elements
• CPC Class Codes: G09G 2340/0435 Change or adaptation of the frame rate of the video stream
• CPC Class Codes: G09G 2340/0464 Positioning
• CPC Class Codes: G09G 2340/0471 Vertical positioning
• CPC Class Codes: G09G 2340/0478 Horizontal positioning
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: G09G 3/007 Use of pixel shift techniques, e.g. by mechanical shift of the physical pixels or by optical shift of the perceived pixels
• CPC Class Codes: G09G 3/3225 using an active matrix

Abstract

Methods for generating and displaying images associated with one or more virtual objects within an augmented reality environment at a frame rate that is greater than a rendering frame rate are described. The rendering frame rate may correspond with the minimum time to render images associated with a pose of a head-mounted display device (HMD). In some embodiments, the HMD may determine a predicted pose associated with a future position and orientation of the HMD, generate a pre-rendered image based on the predicted pose, determine an updated pose associated with the HMD subsequent to generating the pre-rendered image, generate an updated image based on the updated pose and the pre-rendered image, and display the updated image on the HMD. The updated image may be generated via a homographic transformation and/or a pixel offset adjustment of the pre-rendered image by circuitry within the display.

Claims

1. A method, comprising:

loading a rendered image into a buffer of a mobile device;

determining a first transformation for a first portion of the rendered image based on a first predicted pose of the mobile device;

determining a second transformation for a second portion of the rendered image based on a second predicted pose of the mobile device different from the first predicted pose;

generating a first portion of an updated image by applying the first transformation to the first portion of the rendered image;

generating a second portion of the updated image by applying the second transformation to the second portion of the rendered image while the second portion of the rendered image is stored within the buffer; and

displaying the updated image using the mobile device.

2. The method of claim 1, wherein:

the generating the second portion of the updated image includes applying a first pixel offset adjustment to the second portion of the rendered image using the buffer.

3. The method of claim 2, wherein:

the applying the first pixel offset adjustment to the second portion of the rendered image includes shifting the second portion of the rendered image while the second portion of the rendered image is stored within the buffer.

4. The method of claim 1, wherein:

the generating the second portion of the updated image includes applying a first pixel offset adjustment in a first direction to the second portion of the rendered image and applying a second pixel offset adjustment in a second direction to the second portion of the rendered image.

5. The method of claim 4, wherein:

the generating the second portion of the updated image includes shifting the second portion of the rendered image by the first pixel offset adjustment in the first direction and by the second pixel offset adjustment in the second direction while the second portion of the rendered image is stored within the buffer.

6. The method of claim 4, wherein:

the generating the second portion of the updated image includes shifting the second portion of the rendered image by the first pixel offset adjustment in the first direction and by the second pixel offset adjustment in the second direction using the buffer.

7. The method of claim 1, wherein:

the first transformation comprises a first homographic transformation; and

the second transformation comprises a second homographic transformation.

8. The method of claim 1, wherein:

the first portion of the rendered image is associated with a first color field of the rendered image and the second portion of the rendered image is associated with a second color field of the rendered image.

9. The method of claim 1, wherein:

the generating the second portion of the updated image includes generating the second portion of the updated image while the first portion of the updated image is displayed using the mobile device.

10. The method of claim 1, wherein:

the mobile device comprises a head-mounted display device; and

the buffer is integrated within a display of the head-mounted display device.

11. An electronic device, comprising:

a buffer configured to store a rendered image;

one or more processors configured to determine a first transformation for a first portion of the rendered image based on a first predicted pose of the electronic device and determine a second transformation for a second portion of the rendered image based on a second predicted pose of the electronic device different from the first predicted pose, the one or more processors configured generate a first portion of an updated image via application of the first transformation to the first portion of the rendered image and generate a second portion of the updated image via application of the second transformation to the second portion of the rendered image while the second portion of the rendered image is stored within the buffer; and

a display configured to display the updated image.

12. The electronic device of claim 11, wherein:

the one or more processors configured to generate the second portion of the updated image via application of a first pixel offset adjustment to the second portion of the rendered image using the buffer.

13. The electronic device of claim 11, wherein:

the one or more processors configured to shift the second portion of the rendered image while the second portion of the rendered image is stored within the buffer.

14. The electronic device of claim 11, wherein:

the one or more processors configured to apply a first pixel offset adjustment in a first direction to the second portion of the rendered image and apply a second pixel offset adjustment in a second direction to the second portion of the rendered image.

15. The electronic device of claim 14, wherein:

the one or more processors configured to shift the second portion of the rendered image by the first pixel offset adjustment in the first direction and by the second pixel offset adjustment in the second direction while the second portion of the rendered image is stored within the buffer.

16. The electronic device of claim 14, wherein:

the one or more processors configured to shift the second portion of the rendered image by the first pixel offset adjustment in the first direction and by the second pixel offset adjustment in the second direction using the buffer.

17. The electronic device of claim 11, wherein:

the first portion of the rendered image is associated with a first color field of the rendered image and the second portion of the rendered image is associated with a second color field of the rendered image.

18. The electronic device of claim 11, wherein:

the one or more processors configured to generate the second portion of the updated image while the first portion of the updated image is displayed using the display.

19. The electronic device of claim 11, wherein:

the electronic device comprises a head-mounted display device; and

the buffer is integrated within the display of the head-mounted display device.

20. One or more storage devices containing processor readable code for programming one or more processors to perform a method for generating and displaying images using a head-mounted display device, the processor readable code comprising:

processor readable code configured to store a rendered image using a buffer integrated within a display of the head-mounted display device;

processor readable code configured to determine a first homographic transformation for a first portion of the rendered image based on a first predicted pose of the head-mounted display device;

processor readable code configured to determine a second homographic transformation for a second portion of the rendered image based on a second predicted pose of the head-mounted display device different from the first predicted pose;

processor readable code configured to generate a first portion of an updated image via application of the first homographic transformation to the first portion of the rendered image;

processor readable code configured to generate a second portion of the updated image via application of the second homographic transformation to the second portion of the rendered image while the second portion of the rendered image is stored within the buffer; and

processor readable code configured to display the updated image using the display of the head-mounted display device.