Predictive, foveated virtual reality system

• US 10,739,599 B2
• Filed: 06/21/2019
• Issued: 08/11/2020
• Est. Priority Date: 09/22/2016
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Shpunt, Alexander
• Primary Examiner(s): Khan, Ibrahim A
• Application Number:
  US16/448,846
• Publication Number:
  US 20200012106A1
• Time in Patent Office: 417 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0147 comprising a device modifying the resolution of the displayed image
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 13/373 for tracking forward-backward translational head movements, i.e. longitudinal movements
• CPC Class Codes: H04N 13/376 for tracking left-right translational head movements, i.e. lateral movements
• CPC Class Codes: H04N 13/38 for tracking vertical translational head movements
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof

Abstract

A Predictive, Foveated Virtual Reality System may capture views of the world around a user using multiple resolutions. The Predictive, Foveated Virtual Reality System may include one or more cameras configured to capture lower resolution image data for a peripheral field of view while capturing higher resolution image data for a narrow field of view corresponding to a user'"'"'s line of sight. Additionally, the Predictive, Foveated Virtual Reality System may also include one or more sensors or other mechanisms, such as gaze tracking modules or accelerometers, to detect or track motion. A Predictive, Foveated Virtual Reality System may also predict, based on a user'"'"'s head and eye motion, the user'"'"'s future line of sight and may capture image data corresponding to a predicted line of sight. When the user subsequently looks in that direction the system may display the previously captured (and augmented) view.

Claims

1. A system, comprising:

a lower resolution camera configured to capture image data at a lower resolution;

a higher resolution camera configured to capture image data at a higher resolution, wherein image data captured by the higher resolution camera at the higher resolution comprises more pixels per inch image than data captured by the lower resolution camera at the lower resolution, wherein the higher resolution camera is configured to capture image data over a narrower field of view than the lower resolution camera;

a gaze tracking module configured to determine a line of sight for a user of the system;

an adjustable mirror coupled to the gaze tracking module and configured to reflect light along the user'"'"'s line of sight to the higher resolution camera; and

a renderer module configured to render, on a display viewable by the user, an image of a view along the user'"'"'s line of sight, based at least in part on image data captured by the lower resolution camera and image data captured by the higher resolution camera using light reflected by the adjustable mirror.

2. The system of claim 1, wherein the adjustable mirror is configured to reflect light to the higher resolution camera from a plurality of vectors outside a current view of the higher resolution camera.

3. The system of claim 1, further comprising one or more processors configured to adjust the adjustable mirror to reflect light along the user'"'"'s line of sight to the higher resolution camera.

4. The system of claim 1, wherein the gaze tracking module is configured to adjust the mirror to reflect light along the user'"'"'s line of sight to the higher resolution camera.

5. The system of claim 1, wherein to determine the line of sight for the user, the gaze tracking module is configured to detect eye motion for the user.

6. The system of claim 1, wherein the system is further configured to:

determine a predictive line of sight vector representing a potential line of sight for the user at a particular time in the future based, at least in part, on a current movement of the user'"'"'s eyes and a current movement of the user'"'"'s head; and

adjust the mirror to reflect light from the predictive line of sight for the user to the higher resolution camera.

7. The system of claim 6, wherein the processor is further configured to collect image data captured by the higher resolution camera along the predictive line of sight vector prior to the line of sight vector for the user approaching the potential line of sight.

8. The system of claim 6, further comprising one or more accelerometers configured to determine a current movement of the user'"'"'s head based on output of the one or more accelerometers, wherein the predictive line of sight vector is determined based at least in part on the current movement of the user'"'"'s head.

9. The system of claim 6, wherein the system is further configured to determine the particular time in the future based, at least in part, on a length of time required to capture and render to the display image data captured by the lower resolution camera and the higher resolution camera.

10. The system of claim 1, further comprising a head mounted display, wherein the head mounted display comprises the lower resolution camera, the higher resolution camera, the gaze tracking module, the adjustable mirror and the display viewable by the user.

11. A method, comprising:

determining, by a gaze tracking module, a line of sight for a user of a head mounted display;

adjusting an adjustable mirror coupled to the gaze tracking module to reflect light along the line of sight to a higher resolution camera;

capturing, via a lower resolution camera, lower resolution image data corresponding to a lower resolution field of view along the line of sight;

capturing, via the higher resolution camera, higher resolution image data corresponding to a higher resolution field of view along the line of sight, wherein the higher resolution image data comprises a higher pixel density than the lower resolution image data and wherein the higher resolution field of view is narrower than the lower resolution field of view; and

rendering, on a display viewable by the user, an image of a view along the line of sight based at least in part on image data captured by the higher resolution camera using light reflected by the adjustable mirror and based at least in part on image data captured by the lower resolution camera.

12. The method of claim 11, wherein said adjusting comprises adjusting the adjustable mirror to reflect light to the higher resolution camera from a vector outside a current view of the higher resolution camera.

13. The method of claim 11, wherein said adjusting is performed by the gaze tracking module.

14. The method of claim 11, wherein said adjusting is performed by the one or more processors of the head mounted display based on output of the gaze tracking module.

15. The method of claim 11, wherein said determining the line of sight comprises detecting, by the gaze tracking module, eye motion for the user, wherein the line of sight is determined based at least in part on the detected eye motion.

16. The method of claim 11, further comprising:

determining a predictive line of sight vector representing a potential line of sight for a user of a virtual reality system at a particular time in the future based, at least in part, on motion information regarding the user'"'"'s current movement; and

adjusting the adjustable mirror to reflect light from the predictive line of sight to the higher resolution camera.

17. The method of claim 16, further comprising capturing higher resolution image data along the predictive line of sight using light reflected by the adjustable mirror prior to the particular time.

18. The method of claim 16, further comprising determining, by one or more accelerometers of the head mounted display, a current movement of the user'"'"'s head, wherein the predictive line of sight vector is determined based at least in part on the current movement of the user'"'"'s head.

19. The method of claim 16, further comprising determining the particular time in the future based, at least in part, on a length of time required to capture and render image data captured by the lower resolution camera and the higher resolution camera.

20. The method of claim 16, wherein the rendered image is viewable by the user at the particular time.