Display system and method

• US 9,417,452 B2
• Filed: 03/14/2014
• Issued: 08/16/2016
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Schowengerdt, Brian T., Miller, Samuel A.
• Primary Examiner(s): Faulk, Devona
• Assistant Examiner(s): Sun, Hai Tao
• Application Number:
  US14/212,961
• Publication Number:
  US 20140267420A1
• Time in Patent Office: 886 Days
• Field of Search: 345/633
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 1/60 Memory management
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G09G 2320/0266 Reduction of sub-frame artefacts
• CPC Class Codes: G09G 3/007 Use of pixel shift techniques, e.g. by mechanical shift of the physical pixels or by optical shift of the perceived pixels

Abstract

One embodiment is directed to a user display device comprising a housing frame mountable on the head of the user, a lens mountable on the housing frame and a projection sub system coupled to the housing frame to determine a location of appearance of a display object in a field of view of the user based at least in part on at least one of a detection of a head movement of the user and a prediction of a head movement of the user, and to project the display object to the user based on the determined location of appearance of the display object.

Claims

1. A method of operation in a virtual image system or an augmented reality system, the method comprising:

for each of at least some of a plurality of frames being presented to an end user, determining a location of appearance of a virtual object in a field of view of the end user relative to an end user frame of reference;

adjusting a presentation of at least one subsequent frame based at least in part on the determined location of appearance of the virtual object in the field of view of the end users;

predicting an occurrence of a head movement of the end user based at least in part on the determined location of appearance of the virtual object in the field of view of the end user;

estimating at least one value indicative of an estimated speed of the predicted head movement of the end user;

determining at least one first value that at least partially compensates for the estimated speed of the predicted head movement of the end user;

rendering the at least one subsequent frame based at least in part on the determined first value; and

estimating at least one change in a speed in the predicted head movement of the end user, wherein the at least one change in the speed occurs between a start of the predicted head movement and an end of the predicted head movement, and wherein estimating the at least one value indicative of the estimated speed of the predicted head movement includes estimating the at least one value indicative of the estimated speed that at least partially accommodates for the at least one estimated change in the speed in the predicted head movement of the end user.

2. The method of claim 1, wherein the virtual object is newly introduced in the field of view of the end user temporally relative to previous frames presented to the end user.

3. The method of claim 2, wherein the newly introduced virtual object is determined to likely attract an attention of the end user.

4. The method of claim 1 wherein the virtual object is in a new position in the frame relative to a position in at least one previous frame.

5. The method of claim 1, further comprising:

selecting the virtual object based on input indicative of an attention of the end user to the virtual object.

6. The method of claim 5 wherein the input indicative of the attention of the end user to the virtual object is based at least in part on an appearance of the virtual object in a new position as presented to the end user relative to a position of the virtual object as previously presented to the end user.

7. The method of claim 5 wherein the input indicative of the attention of the end user to the virtual object is based at least in part on how quickly a position of the virtual object as presented to the end user changes relative to the position of the virtual object as previously presented to the end user.

8. The method of claim 1 wherein the virtual object is in a new position as presented to the end user relative to a previous position of the virtual object as previously presented to the end user.

9. The method of claim 1 wherein the adjusting of the presentation of the at least one subsequent frame includes presenting the at least one subsequent frame with a center of the at least one subsequent frame shifted toward, but not centered on, the determined location of appearance of the virtual object in the field of view of the end user.

10. The method of claim 1, wherein the adjusting of the presentation of the at least one subsequent frame includes presenting the at least one subsequent frame with a center of the at least one subsequent frame shifted to the determined location of appearance of the virtual object in the field of view of the end user.

11. The method of claim 1 wherein estimating at least one change in the speed in the predicted head movement of the end user includes estimating the at least one change between a first defined time after the start of the predicted head movement and a second defined time before the end of the predicted head movement .

12. The method of claim 1, further comprising:

estimating at least one value indicative of an estimated acceleration of the predicted head movement of the end user;

determining at least one second value that at least partially compensates for the estimated acceleration of the predicted head movement of the end user; and

rendering the at least one subsequent frame based at least in part on the determined second value.

13. The method of claim 1, further comprising:

receiving information indicative of an identity of the end user; and

retrieving at least one user specific historical attribute for the end user based on the received information indicative of the identity of the end user, wherein the user specific historical attribute is indicative of at least one of a previous head movement speed for the end user, a previous head movement acceleration for the end user, and a previous eye movement to head movement relationship for the end user.

14. The method of claim 1 wherein the virtual object is at least one of a virtual text object, a virtual numeric object, a virtual alphanumeric object, a virtual tag object, a virtual field object, a virtual chart object, a virtual map object, a virtual instrumentation object or a virtual visual representation of a physical object.