Virtual/augmented reality system having dynamic region resolution

• US 10,176,639 B2
• Filed: 07/15/2016
• Issued: 01/08/2019
• Est. Priority Date: 11/27/2014
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Schowengerdt, Brian T., Edwin, Lionel Ernest, Yeoh, Ivan L., Schuelke, Aaron Mark, Miller, Samuel A.
• Primary Examiner(s): Lhymn, Sarah
• Application Number:
  US15/212,053
• Publication Number:
  US 20160328884A1
• Time in Patent Office: 907 Days
• Field of Search: 345619, 345632, 345633
• CPC Class Codes: G02B 17/08 Catadioptric systems used in non-imaging applications G02B19/00
• CPC Class Codes: G02B 2006/0098 for scanning
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0147 comprising a device modifying the resolution of the displayed image
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 6/00 Light guides; Structural details of arrangements comprising light guides and other optical elements, e.g. couplings
• CPC Class Codes: G02B 6/0008 the light being emitted at the end of the fibre
• CPC Class Codes: G02B 6/02042 Multicore optical fibres
• CPC Class Codes: G02B 6/06 the relative position of the fibres being the same at both ends, e.g. for transporting images
• CPC Class Codes: G02B 6/32 having lens focusing means positioned between opposed fibre ends with lens being an integral part of the single fibre end G02B6/262
• CPC Class Codes: G02B 6/34 utilising prism or grating G02B6/293 takes precedence
• CPC Class Codes: G02B 6/40 having fibre bundle mating means
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/14 : Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 3/18 : Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 3/40 : Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G09G 2340/0407 : Resolution change, inclusive of the use of different resolutions for different screen areas
• CPC Class Codes: G09G 2340/14 : Solving problems related to the presentation of information to be displayed
• CPC Class Codes: G09G 2350/00 : Solving problems of bandwidth in display systems
• CPC Class Codes: G09G 2354/00 : Aspects of interface with display user
• CPC Class Codes: G09G 3/025 : with scanning or deflecting the beams in two directions or dimensions
• CPC Class Codes: G09G 5/00 : Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T
• CPC Class Codes: H04N 13/344 : with head-mounted left-right displays
• CPC Class Codes: H04N 13/383 : for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 13/398 : Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 23/45 : for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 5/74 : Projection arrangements for image reproduction, e.g. using eidophor

Abstract

A virtual image generation system and method is provided. A plurality of synthetic image frames of the three-dimensional scene are rendered, and sequentially displayed to an end user. Each of the displayed image frames has a non-uniform resolution distribution.

Claims

1. A method of operating a virtual image generation system, the method comprising:

estimating a focal point of an eye within a field of view of the end user;

rendering a plurality of synthetic image frames of a three-dimensional scene;

sequentially displaying the plurality of image frames to the end user;

generating a non-uniform resolution distribution for each of the displayed image frames in response to the estimated focal point, the non-uniform resolution distribution having a region of highest resolution and a region of lower resolution, wherein the region of highest resolution is coincident with the estimated focal point, and wherein the estimated focal point of the end user has an error margin to provide a focal range within the field of the view of the end user, and the region of highest resolution intersects the focal range;

blurring the displayed image frames in the region of lower resolution; and

dynamically modifying the error margin based on an assumed eye angular velocity profile.

2. The method of claim 1, wherein each of the image frames is rendered with the non-uniform resolution distribution.

3. The method of claim 1, wherein each of the image frames is rendered having a uniform resolution distribution, and wherein displaying the respective image frame comprises incorporating the non-uniform resolution distribution into the already rendered image frame.

4. The method of claim 1, wherein the region of highest resolution and a region of lower resolution are discrete.

5. The method of claim 4, further comprising selecting the region of highest resolution from a field of view template having a plurality of discrete regions.

6. The method of claim 5, wherein the plurality of discrete regions in the field of view template overlap each other.

7. The method of claim 5, wherein each of the image frames is scanned in a spiral pattern, and wherein each of the discrete regions is sector-shaped.

8. The method of claim 1, wherein each of the displayed image frames is scanned, and blurring the displayed image frames comprises dithering scan lines in adjacent displayed image frames in the region of lower resolution.

9. The method of claim 1, wherein blurring the displayed image frames comprises defocusing the displayed image frames in the region of lower resolution.

10. The method of claim 1, wherein each of the displayed image frames is scanned, and the blurring of the image frames in the region of lower resolution minimizes visual artifacts due to relatively large spacing between scan lines in the region of lower resolution.

11. The method of claim 1, wherein a resolution distribution of each of the displayed image frames has a slope that matches the slope of an acuity distribution of an eye of the end user.

12. The method of claim 1, wherein a resolution distribution of each of the displayed image frames has a slope that is greater than the slope of an acuity distribution of an eye of the end user.

13. A virtual image generation system for use by an end user, comprising:

memory storing a three-dimensional scene;

a control subsystem configured for

rendering a plurality of synthetic image frames of the three-dimensional scene,

estimating a focal point of an eye within a field of view of the end user, and generating a non-uniform resolution distribution for each of the displayed image frames based on the estimated focal point, the non-uniform resolution distribution having a region of highest resolution and a region of lower resolution, wherein the region of highest resolution is coincident with the estimated focal point, and wherein the estimated focal point of the end user has an error margin to provide a focal range within the field of the view of the end user, and the region of highest resolution intersects the focal range,

blurring the displayed image frames in the region of lower resolution, and

dynamically modifying the error margin based on an assumed eye angular velocity profile; and

a display subsystem configured for sequentially displaying the plurality of image frames to the end user, wherein each of the displayed image frames has a non-uniform resolution distribution.

14. The virtual image generation system of claim 13, wherein the display subsystem includes a projection subsystem and a partially transparent display surface, the projection subsystem configured for projecting the image frames onto the partially transparent display surface, and the partially transparent display surface is configured for being positioned in the field of view between the eyes of the end user and an ambient environment.

15. The virtual image generation system of claim 13, wherein the control subsystem is configured for rendering each of the image frames with the non-uniform resolution distribution.

16. The virtual image generation system of claim 13, wherein the control subsystem is configured for rendering each of the image frames with a uniform resolution distribution, and wherein the display subsystem is configured for displaying the respective image frame by incorporating the non-uniform resolution distribution into the already rendered image frame.

17. The virtual image generation system of claim 13, wherein the region of highest resolution and a region of lower resolution are discrete.

18. The virtual image generation system of claim 17, wherein the control subsystem is configured for selecting the region of highest resolution from a field of view template having a plurality of discrete regions.

19. The virtual image generation system of claim 13, further comprising one or more sensors configured for detecting the focal point of the end user, wherein the control subsystem is configured for estimating the focal point from the detected focal point.