REPROJECTION OLED DISPLAY FOR AUGMENTED REALITY EXPERIENCES

• US 20150002542A1
• Filed: 06/28/2013
• Published: 01/01/2015
• Est. Priority Date: 06/28/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Margolis, Jeffrey Neil, Shetter, Martin, Michail, Ashraf Ayman, Corlett, Barry, Chan, Calvin, Pearson, Andrew
• Granted Patent: US 9,443,355 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 3/02 Affine transformations for image registration G06T3/147; for image mosaicing G06T3/4038
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G09G 2300/0804 Sub-multiplexed active matrix panel, i.e. wherein one active driving circuit is used at pixel level for multiple image producing elements
• CPC Class Codes: G09G 2340/0435 Change or adaptation of the frame rate of the video stream
• CPC Class Codes: G09G 2340/0464 Positioning
• CPC Class Codes: G09G 2340/0471 Vertical positioning
• CPC Class Codes: G09G 2340/0478 Horizontal positioning
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: G09G 3/007 Use of pixel shift techniques, e.g. by mechanical shift of the physical pixels or by optical shift of the perceived pixels
• CPC Class Codes: G09G 3/3225 using an active matrix

Abstract

Methods for generating and displaying images associated with one or more virtual objects within an augmented reality environment at a frame rate that is greater than a rendering frame rate are described. The rendering frame rate may correspond with the minimum time to render images associated with a pose of a head-mounted display device (HMD). In some embodiments, the HMD may determine a predicted pose associated with a future position and orientation of the HMD, generate a pre-rendered image based on the predicted pose, determine an updated pose associated with the HMD subsequent to generating the pre-rendered image, generate an updated image based on the updated pose and the pre-rendered image, and display the updated image on the HMD. The updated image may be generated via a homographic transformation and/or a pixel offset adjustment of the pre-rendered image by circuitry within the display.

Claims

1. A mobile device, comprising:

one or more processors, the one or more processors determine a predicted pose associated with the mobile device, the predicted pose corresponds with a first point in time, the one or more processors generate a rendered image based on the predicted pose, the one or more processors determine an updated pose associated with the mobile device subsequent to determining the predicted pose, the updated pose corresponds with the first point in time; and

a see-through display in communication with the one or more processors, the see-through display generates an updated image based on the updated pose and the rendered image, the see-through display displays the updated image.

2. The mobile device of claim 1, wherein:

the see-through display generates the updated image by applying a pixel offset adjustment to at least a portion of the rendered image.

3. The mobile device of claim 2, wherein:

the see-through display applies the pixel offset adjustment by loading a first image associated with the rendered image into a pixel array of the see-through display and shifting the first image within the pixel array.

4. The mobile device of claim 2, wherein:

the see-through display applies the pixel offset adjustment by loading a first image associated with the rendered image into a buffer of the see-through display and shifting the first image within the buffer.

5. The mobile device of claim 1, wherein:

the see-through display generates the updated image by applying a homographic transformation to at least a portion of the rendered image.

6. The mobile device of claim 5, wherein:

the homographic transformation comprises an affine transformation.

7. The mobile device of claim 1, wherein:

the mobile device comprises an HMD; and

the see-through display comprises an OLED display.

8. A method for displaying images, comprising:

determining a predicted pose associated with a mobile device, the predicted pose corresponds with a first point in time;

generating a rendered image based on the predicted pose;

determining an updated pose associated with the mobile device subsequent to the generating a rendered image, the updated pose corresponds with the first point in time;

determining a pose difference between the predicted pose and the updated pose;

generating an updated image based on the pose difference; and

displaying the updated image on a display of the mobile device.

9. The method of claim 8, wherein:

the generating an updated image includes applying a pixel offset adjustment to at least a portion of the rendered image.

10. The method of claim 9, wherein:

the applying a pixel offset adjustment includes loading a first image associated with the rendered image into a pixel array of the display and shifting the first image within the pixel array.

11. The method of claim 9, wherein:

the applying a pixel offset adjustment includes loading a first image associated with the rendered image into a buffer of the display and shifting the first image within the buffer.

12. The method of claim 8, wherein:

the generating an updated image includes applying a homographic transformation to at least a portion of the rendered image.

13. The method of claim 12, wherein:

the homographic transformation comprises an affine transformation.

14. The method of claim 8, further comprising:

acquiring a pose history associated with the mobile device; and

determining a current pose of the mobile device prior to the determining a predicted pose, the determining a predicted pose includes determining the predicted pose based on the current pose and the pose history.

15. The method of claim 14, wherein:

the pose history includes an acceleration and a velocity of the mobile device prior to determining the predicted pose.

16. The method of claim 8, wherein:

the mobile device comprises an HMD; and

the display comprises an OLED display.

17. One or more storage devices containing processor readable code for programming one or more processors to perform a method for displaying images associated with one or more virtual objects using an HMD comprising the steps of:

acquiring a pose history associated with the HMD, the pose history corresponds with a time period prior to a first point in time;

determining a current pose associated with the HMD corresponding with the first point in time;

determining a predicted pose associated with the HMD corresponding with a second point in time subsequent to the first point in time, the determining a predicted pose includes determining the predicted pose based on the current pose and the pose history;

generating a rendered image based on the predicted pose;

determining an updated pose associated with the HMD corresponding with the second point in time;

generating an updated image based on the updated pose and at least a portion of the rendered image; and

displaying the updated image on a display of the HMD.

18. The one or more storage devices of claim 17, wherein:

the generating an updated image includes applying a pixel offset adjustment to at least a portion of the rendered image.

19. The one or more storage devices of claim 18, wherein:

the applying a pixel offset adjustment includes loading a first image associated with the rendered image into a pixel array of the display and shifting the first image within the pixel array.

20. The one or more storage devices of claim 17, wherein:

the generating an updated image includes applying a homographic transformation to at least a portion of the rendered image using circuitry integrated with the display.