Over-rendering techniques in augmented or virtual reality systems

• US 10,510,188 B2
• Filed: 05/06/2015
• Issued: 12/17/2019
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Schowengerdt, Brian T., Miller, Samuel A.
• Primary Examiner(s): Sun, Hai Tao
• Application Number:
  US14/704,981
• Publication Number:
  US 20150235417A1
• Time in Patent Office: 1,686 Days
• Field of Search: 345633
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 1/60 Memory management
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G09G 2320/0266 Reduction of sub-frame artefacts
• CPC Class Codes: G09G 3/007 Use of pixel shift techniques, e.g. by mechanical shift of the physical pixels or by optical shift of the perceived pixels

Abstract

One embodiment is directed to a user display device comprising a housing frame mountable on the head of the user, a lens mountable on the housing frame and a projection sub system coupled to the housing frame to determine a location of appearance of a display object in a field of view of the user based at least in part on at least one of a detection of a head movement of the user and a prediction of a head movement of the user, and to project the display object to the user based on the determined location of appearance of the display object.

Claims

1. A method of operation in a virtual image presentation system, the method comprising:

over-rendering a frame of a sequence of frames for a field of view provided by the virtual image presentation system such that a set of total pixels included in the frame exceeds a first set of pixels included in a reduced frame that is to be presented in a maximum area of a display to a user when the display is configured to present one or more virtual images corresponding to the frame at a maximum image resolution;

determining a detected or predicted speed or a detected or predicted acceleration of head movement of the user;

predicting, with a predictive head tracking module including a processor and one or more transducers in the virtual image presentation system, at least one predicted head movement of the user based at least in part upon the detected or predicted speed or the detected or predicted acceleration of head movement of the user;

determining a portion of the frame to present to the user based on at least one of a detected head movement and the at least one predicted head movement, wherein

the portion is of a size that is smaller than an entire size of the frame, and

the portion of the frame to be presented to the user is based at least in part on determining a location of a virtual object having at least a defined minimum speed in the field of view of the user;

selectively reading out data for the portion of the frame from at least one frame buffer of the virtual image presentation system;

adjusting an actual or perceived pixel size of at least one pixel of a set of pixels in a first portion of at least one subsequent frame into an adjusted pixel size of the at least one pixel based in part or in whole upon a variation in pixel spacing values of adjacent pixels and the at least one predicted head movement, wherein a pixel spacing value indicates spacing between two adjacent pixels and is predicted to cause the variation in the first portion relative to a remaining portion of the at least one subsequent frame based at least in part upon the at least one predicted head movement; and

presenting the at least one subsequent frame after the frame to the user at least by using at least the adjusted pixel size for the first portion in the at least one subsequent frame and by using the actual or perceived pixel size for the remaining portion in the at least one subsequent frame.

2. The method of claim 1, wherein the portion of the frame to be presented to the user is based at least in part on the determining the location of the virtual object in the field of view.

3. The method of claim 1, wherein the portion of the frame to be presented to the user is based at least in part on determining the location of the virtual object when newly introduced in the field of view of the user temporally relative to one or more previous frames presented to the user.

4. The method of claim 1, wherein the portion of the frame to be presented to the user is based at least in part on determining the location of the virtual object in a new position in the frame relative to a position in at least one previous frame.

5. The method of claim 1, further comprising:

selecting the virtual object based at least in part on an input indicative of attention of the user to the virtual object, wherein the portion of the frame to be presented to the user is based on the virtual object.

6. The method of claim 5, wherein the input indicative of the attention of the user is based at least in part on how quickly a position of the virtual object as presented to the user changes relative to the position of the virtual object as previously presented to the user.

7. The method of claim 1, further comprising reading the portion out of the at least one frame buffer for the at least one subsequent frame that shifts a center of the subsequent frame toward the portion of the frame to present.

8. The method of claim 1, wherein only the portion is selectively read out, and the at least one predicted head movement is further predicted by using historical data or attributes of the user.

9. The method of claim 1, wherein the reading out comprises driving a light source to one or more optical fibers.

10. The method of claim 1, further comprising adjusting a perceived pixel size of at least one pixel of a first plurality of pixels relative to a second plurality of pixels based at least in part upon the detected or predicted speed or the detected or predicted acceleration of the head movement of the user, without using fly pixels-based display techniques.

11. The method of claim 1, further comprising:

determining boundaries based in part or in whole upon a current location in an image and a predicted location in a subsequent image to be presented to the user; and

shifting, relative to the current location determined by an eye-tracking module in the virtual image presentation system, the subsequent image that is to be presented to the user with one or more over-rendered frames.

12. The method of claim 1, wherein the portion is determined based in part or in whole on one or more factors, and the one or more factors comprise at least one of a first location in an image to which the user is paying attention or is predicted to pay attention, a second location in the image on which the user focuses or is predicted to focus, one or more newly appearing virtual objects presented to the user, one or more rapidly moving virtual objects, visual attractiveness of the virtual object, a previously designated virtual object, or a combination thereof.

13. The method of claim 1, further comprising:

identifying a plurality of resolutions that are supported by the virtual image presentation system;

determining an upper end resolution from the plurality of resolutions; and

identifying the field of view for the virtual image presentation system.

14. The method of claim 13, further comprising:

shifting boundaries of images presented to the user based in part or in whole upon a size of the frame;

configuring one or more frame buffers based in part or in whole upon the upper end resolution and the field of view; and

successively buffering a plurality of over rendered frames in the one or more frame buffers in the virtual image presentation system.

15. The method of claim 1, further comprising:

detecting, with an eye tracking module, eye tracking information of at least one eye of the user;

determining an indication of attention or focus of the user based in part or in whole upon the eye tracking information;

associating the virtual object or a virtual event with the indication of the attention or the focus of the user;

predicting a change in the focus of the user; and

determining the at least one predicted head movement based in part or in whole upon a shift in focus of the user.

16. A virtual image presentation system, comprising:

a projector device configured to project virtual contents;

a processor configured to over-render a frame of a sequence of frames for a field of view provided by the virtual image presentation system such that a total number of pixels included in the frame exceeds a pixel count for a maximum area of display when the display is configured to display virtual contents at a maximum resolution;

a predictive head tracking module including the processor and one or more transducers that generate head tracking data and configured to determine a detected or predicted speed or a detected or predicted acceleration of head movement of a user and to predict at least one predicted head movement of a user relative to a nominal value based at least in part upon the detected or predicted speed or the detected or predicted acceleration of head movement;

the processor further configured to determine a portion of the frame to present to the user based on at least one of a detected head movement and the at least one predicted head movement and to selectively read out data for the portion of the frame from at least one frame buffer of the virtual image presentation system, wherein

the portion is of a size that is smaller than an entire size of the frame, and

the portion of the frame to be presented to the user is based at least in part on determining a location of a virtual object having at least a defined minimum speed in the field of view of the user;

the processor further configured to adjust an actual or perceived pixel size of at least one pixel of a set of pixels in a first portion of at least one subsequent frame into an adjusted pixel size of the at least one pixel based in part or in whole upon a variation in pixel spacing values of adjacent pixels, wherein a pixel spacing value indicates spacing between two adjacent pixels and is predicted to cause the variation in the first portion relative to a remaining portion of the at least one subsequent frame based at least in part upon the at least one predicted head movement; and

the processor further configured to present the at least one subsequent frame after the frame to the user at least by using at least the adjusted pixel size for the first portion in the at least one subsequent frame and by using the actual or perceived pixel size for the remaining portion in the at least one subsequent frame.

17. The virtual image presentation system of claim 16, wherein the projector device is configured to move in at least one degree of freedom and comprises a piezoelectric collar that supports a projector element in the projector device and an actuator operatively coupled to the piezoelectric collar.

18. The virtual image presentation system of claim 16, wherein the predictive head tracking module-comprises:

a light source configured to project light in a first geometric shape into at least one eye of the user; and

a reflected light receiver configured to receive reflected light in a second geometric shape from the at least one eye of the user.

19. The virtual image presentation system of claim 18, wherein the predictive head tracking module is further configured to determine one or more movement characteristics of the at least one eye of the user based in part or in whole upon the first geometric shape and the second geometric shape.