Blanking techniques in augmented or virtual reality systems

• US 10,304,246 B2
• Filed: 05/06/2015
• Issued: 05/28/2019
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Schowengerdt, Brian T., Miller, Samuel A.
• Primary Examiner(s): Sun, Hai Tao
• Application Number:
  US14/704,976
• Publication Number:
  US 20150235452A1
• Time in Patent Office: 1,483 Days
• Field of Search: 345633
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 1/60 Memory management
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G09G 2320/0266 Reduction of sub-frame artefacts
• CPC Class Codes: G09G 3/007 Use of pixel shift techniques, e.g. by mechanical shift of the physical pixels or by optical shift of the perceived pixels

Abstract

One embodiment is directed to a user display device comprising a housing frame mountable on the head of the user, a lens mountable on the housing frame and a projection sub system coupled to the housing frame to determine a location of appearance of a display object in a field of view of the user based at least in part on at least one of a detection of a head movement of the user and a prediction of a head movement of the user, and to project the display object to the user based on the determined location of appearance of the display object.

Claims

1. A method of presenting virtual contents with a virtual or augmented image presentation system, the method comprising:

displaying at least one virtual object to an end user at least by projecting light rays corresponding to the at least one virtual object to at least one eye of the end user with a projector subsystem comprising one or more projectors;

identifying a plurality of relations between eye movements and head movements in a plurality of pairs of respective directions, a relation in the plurality of relations indicating a respective extent of the eye movements along a respective pair of directions of the plurality of pairs of directions before initiation of the head movements in the respective direction;

determining a distance between a current focus of the end user and a determined location of the at least one virtual object, wherein the distance includes an angular distance between the current focus of the end user and the determined location of the at least one virtual object;

predicting a predicted head movement by the end user based in part or in whole upon the distance between the current focus of the end user and the determined location of the at least one virtual object and further upon the plurality of relations between the eye movements and the head movements in the plurality of pairs of directions;

temporarily blanking at least a portion of a presentation of the at least one virtual object for a refresh period on a display device based at least in part on the predicted head movement and a detected head movement captured by at least one transducer or sensor, wherein the refresh period represents a temporal period between a first end of a scan line or scan pattern and a first start of a temporally successive scan line or scan pattern or between a second end of a frame and a second start of a temporally successive frame;

pulsing or modulating one or more light sources producing one or more spots or dots of light at a first frame rate that is lower than a second frame rate of one or more image capturing devices of the virtual or augmented image presentation system;

capturing, with at least one sensor, the one or more spots or dots as one or more line traces, wherein a line trace comprises a line direction and a line length;

determining a direction of the detected head movement based at least in part upon the line direction; and

determining a velocity of the detected head movement based at least in part upon the line length.

2. The method of claim 1, further comprising:

processing head tracking data supplied via the at least one transducer or sensor to determine the predicted head movement, wherein the head tracking data comprises at least an orientation of a head of the end user;

for each image of at least some of images presented to the end user, determining a location of appearance of the at least one virtual object in a field of view of the end user relative to an end user frame of reference;

generating assessment results at least by assessing whether the location requires the end user to turn a head of the end user; and

predicting the head movement based at least in part on the assessment results.

3. The method of claim 1, further comprising:

determining a location of appearance of the at least one virtual object when the at least one virtual object is newly introduced in a field of view of the end user;

generating assessment results at least by assessing whether the location requires the end user to turn a head of the end user; and

predicting the head movement based at least in part on the assessment results.

4. The method of claim 1, further comprising:

determining a location of appearance of the at least one virtual object in a new position in an image relative to a position of the at least one virtual object in at least one previous image;

generating assessment results at least by assessing whether the location requires the end user to turn a head of the end user; and

predicting the head movement based in part or in whole on the assessment results.

5. The method of claim 1, wherein temporarily blanking at least the portion of the at least one virtual object includes strobing a display during at least a portion of a head movement or predicted head movement.

6. The method of claim 1, wherein temporarily blanking at least the portion of the at least one virtual object includes strobing a backlight of a display during at least a portion of a head movement or the predicted head movement.

7. The method of claim 1, wherein the predicted head movement is based at least in part on determining that the predicted head movement value exceeds a predicted head movement value.

8. The method of claim 1, further comprising displaying an adjusted virtual image based at least in part on the predicted head movement.

9. The method of claim 1, further comprising:

acquiring one or more images of at least a portion an environment in which the end user is located; and

determining the predicted head movement based in part or in whole upon the one or more images.

10. The method of claim 1, further comprising:

predicting the predicted head movement based in part or in whole upon at least one of a first determination of where one or more new virtual objects appear in a field of view, a second determination of where the virtual object moves, or a third determination of one or more locations of one or more corresponding attractive virtual objects in the field of view.

11. The method of claim 1, further comprising:

acquiring head movement data at one or more transducers or sensors;

determining one or more averages or integrations of a speed, an acceleration, or a range of head movement of the end user over a plurality of time points; and

predicting the predicted head movement based in part or in whole upon the one or more averages or integrations.

12. The method of claim 1, further comprising:

acquiring head movement data at one or more transducers or sensors; and

determining, at a microprocessor or a controller subsystem, a location of the virtual object.

13. The method of claim 12, further comprising:

determining, at the microprocessor or the controller subsystem, relative visual attractiveness of the virtual object based in part or in whole upon one or more factors, wherein the one or more factors include a motion characteristic of the virtual object, one or more geometric attributes of the virtual object, or one or more optical characteristics of the virtual object.

14. The method of claim 12, further comprising:

determining a range of eye movement for one or more users including the end user;

identifying a current focus of at least one eye of the end user;

determining a distance between the current focus and the location of the at least one virtual object;

identifying information comprising the relation between the eye movement and the head movement of one or more users including the end user in one or more directions; and

generating determination results by determining whether the location of the at least one virtual object results in a change in an orientation of a head of the end user based in part or in whole upon the relation.

15. The method of claim 14, further comprising:

determining an extent of the change in the orientation of a head the end user; and

predicting the predicted head movement based in part or in whole upon the extent of the change and the determination results.

16. The method of claim 15, further comprising:

identifying one or more scan patterns that present the at least one virtual object as a plurality of pixels in a sequence for a frame to the end user; and

temporarily blanking the at least the portion of the virtual object at least by presenting at least a portion of a white frame in place of the frame to the end user.

17. A virtual or augmented image presentation system, comprising:

a projector subsystem comprising one or more projectors and configured to display at least one virtual object to an end user at least by projecting light rays corresponding to the at least one virtual object to at least one eye of the end user with the one or more projectors;

a processor configured to;

identify a plurality of relations between eye movements and head movements in a plurality of pairs of respective directions, a relation in the plurality of relations indicating a respective extent of the eye movements along a respective pair of directions of the plurality of pairs of directions before initiation of the head movements in the first direction;

determine a distance between a current focus of the end user and a determined location of the at least one virtual object, wherein the distance includes an angular distance between the current focus of the end user and the determined location of the at least one virtual object;

predict a predicted head movement by the end user based in part or in whole upon the distance between the current focus of the end user and the determined location of the at least one virtual object and further upon the plurality of relations between eye movement and the head movements in the plurality of pairs of directions;

a control subsystem including a non-transitory processor-readable medium and one or more integrated circuits and operatively coupled to the processor and the projector subsystem to temporarily blank at least a presentation of a portion of the at least one virtual object for a refresh period on a display device based at least in part on a predicted head movement, wherein the refresh period represents a temporal period between a first end of a scan line or scan pattern and a first start of a temporally successive scan line or scan pattern or between a second end of a frame and a second start of a temporally successive frame;

one or more light sources configured to pulse or modulate to produce one or more spots or dots of light at a first frame rate that is lower than a second frame rate of one or more image capturing devices of the virtual or augmented image presentation system; and

at least one sensor configured to capture the one or more spots or dots as one or more line traces, wherein a line trace comprises a line direction and a line length,

where the processor is also configured to determine a direction of the detected head movement based at least in part upon the line direction and to determine a velocity of the detected head movement based at least in part upon the line length.

18. The virtual or augmented image presentation system of claim 17, wherein the projector subsystem further comprises:

a support structure comprising a piezoelectric collar that is configured to accommodate at least the one or more projectors and to allow at least a part of the one or more projectors to oscillate with one or more resonance frequencies;

one or more actuators configured to move at least the part of the one or more projectors in the projector subsystem; and

a controller operatively coupled to the one or more actuators and configured to determine a location of appearance of the at least one virtual object based in part or in whole upon head tracking data received from one or more transducers or one or more sensors of the virtual or augmented image presentation system.