Predictive, foveated virtual reality system

• US 10,330,935 B2
• Filed: 09/21/2017
• Issued: 06/25/2019
• Est. Priority Date: 09/22/2016
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Shpunt, Alexander
• Primary Examiner(s): Khan, Ibrahim A
• Application Number:
  US15/711,933
• Publication Number:
  US 20180081178A1
• Time in Patent Office: 642 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0147 comprising a device modifying the resolution of the displayed image
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 13/373 for tracking forward-backward translational head movements, i.e. longitudinal movements
• CPC Class Codes: H04N 13/376 for tracking left-right translational head movements, i.e. lateral movements
• CPC Class Codes: H04N 13/38 for tracking vertical translational head movements
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof

Abstract

A Predictive, Foveated Virtual Reality System may capture views of the world around a user using multiple resolutions. The Predictive, Foveated Virtual Reality System may include one or more cameras configured to capture lower resolution image data for a peripheral field of view while capturing higher resolution image data for a narrow field of view corresponding to a user'"'"'s line of sight. Additionally, the Predictive, Foveated Virtual Reality System may also include one or more sensors or other mechanisms, such as gaze tracking modules or accelerometers, to detect or track motion. A Predictive, Foveated Virtual Reality System may also predict, based on a user'"'"'s head and eye motion, the user'"'"'s future line of sight and may capture image data corresponding to a predicted line of sight. When the user subsequently looks in that direction the system may display the previously captured (and augmented) view.

Claims

1. A system, comprising:

one or more lower resolution cameras configured to capture image data at a first lower resolution;

at least one higher resolution camera configured to capture image data at a second higher resolution, wherein image data captured at the second higher resolution comprises more pixels per inch image data captured at the first lower resolution;

a display viewable by a user of the system;

a processor and a memory comprising program instruction that when executed by the processor cause the processor to;

determine a view direction based, at least in part, on a line of sight vector for the user;

render, on the display, an image comprising lower resolution image data captured by the one or more lower resolution cameras and higher resolution image data captured by the at least one higher resolution camera, wherein the higher resolution image data comprises image data captured by the at least one higher resolution camera along the line of sight vector and wherein the lower resolution image data is captured by the one or more lower resolution cameras within a field of view comprising the line of sight vector and based on a current position of the user'"'"'s head.

2. The system of claim 1, further comprising one or more gaze tracking sensors configured to track eye movements of the user'"'"'s eyes, wherein the line of sight vector is determined based, at least in part, on output of the one or more gaze tracking sensors.

3. The system of claim 2, wherein the at least one higher resolution camera comprises one or more mirrors configured to reflect light from a plurality of vectors based on the output of the one or more eye tracking sensors.

4. The system of claim 1, wherein the processor is further configured to:

determine a predictive line of sight vector representing a potential line of sight for the user at a particular time in the future based, at least in part, on a current movement of the user'"'"'s eyes and a current movement of the user'"'"'s head.

5. The system of claim 4, further comprising:

one or more accelerometers configured to determine the current movement of the user'"'"'s head; and

one or more eye tracking sensors configured to determine the current movement of the user'"'"'s eyes.

6. The system of claim 4, wherein the processor is further configured to:

determine the particular time in the future based, at least in part, on a length of time required to capture and render to the display image data captured by the one or more lower resolution cameras and by the at least one higher resolution camera.

7. The system of claim 4, wherein the processor is further configured to:

collect image data captured by the at least one higher resolution camera along the predictive line of sight vector prior to the line of sight vector for the user approaching the potential line of sight.

8. The system of claim 7, wherein the processor is further configured to:

render, to the display in response to a determination that the line of sight vector approaches the predictive line of sight, a view comprising the image data captured along the predictive line of sight vector.

9. The system of claim 7, wherein the at least one higher resolution camera is configured to capture image data at a plurality of vectors different from the line of sight vector for the user.

10. The system of claim 7, wherein the at least one higher resolution camera comprises one or more mirrors configured to direct light from a plurality of vectors to a sensor of the at least one higher resolution camera, wherein to capture image data along the predictive line of sight vector, the at least one camera is configured to capture image data via the one or more mirrors.

11. The system of claim 7, wherein the at least one higher resolution camera comprises a moveable lens, wherein to capture image data along the predictive line of sight vector, the at least one camera is configured to move the lens in line with the predictive line of sight vector.

12. A system, comprising:

one or more cameras configured to capture image data;

a display viewable by a user of the system;

a processor and a memory comprising program instruction that when executed by the processor cause the processor to;

determine a predictive line of sight vector representing a potential line of sight for the user at a particular time in the future based, at least in part, on a current movement vector for the user;

capture image data, via the one or more cameras, along the predictive line of sight vector prior to a current line of sight for the user approaching the potential line of sight; and

render on the display, in response to the user'"'"'s line of sight approach the predictive line of sight vector, a view comprising the image data captured along the predictive line of sight vector.

13. The system of claim 12, further comprising one or more gaze tracking sensors configured to track eye movements of the user'"'"'s eyes, wherein the current movement vector for the user comprises a current eye movement based on output of the one or more gaze tracking sensors.

14. The system of claim 12, further comprising one or more accelerometers configured to determine the current movement of the user'"'"'s head, wherein the current movement vector for the user comprises a current head movement based on output of the one or more accelerometers.

15. The system of claim 12, wherein the program instructions further cause the processor to determine the particular time in the future based, at least in part, on a length of time required to capture and render to the display image data captured by the one or more cameras.

16. The system of claim 12, wherein the one or more cameras comprise a plurality of cameras comprising:

one or more lower resolution cameras configured to capture image data at a first lower resolution;

at least one higher resolution camera configured to capture image data at a second higher resolution, wherein image data captured at the second higher resolution comprises more pixels per inch image data captured at the first lower resolution; and

wherein to capture image data along the predictive line of sight vector, the processor is further configured to collect image data captured by the at least one higher resolution camera along the predictive line of sight vector prior to line of sight vector for the user approaching the potential line of sight.

17. The system of claim 12, wherein at least one camera of the one or more cameras comprises one or more mirrors configured to direct light from a plurality of vectors to a sensor of the at least one camera, wherein to capture image data along the predictive line of sight vector, the at least one camera is configured to capture image data via the one or more mirrors.

18. The system of claim 12, wherein at least one camera of the one or more cameras comprises a moveable lens, wherein to capture image data along the predictive line of sight vector, the at least one camera is configured to move the lens in line with the predictive line of sight vector.

19. A method, comprising:

determining a predictive line of sight vector representing a potential line of sight for a user of a virtual reality system at a particular time in the future based, at least in part, on motion information regarding the user'"'"'s current movement;

capturing, via one or more lower resolution cameras, lower resolution image data corresponding to a lower resolution field of view along the predictive line of sight vector;

capturing, via one or more higher resolution cameras, higher resolution image data corresponding to a higher resolution field of view along the predictive line of sight vector, wherein the higher resolution image data comprises a higher pixel density than the lower resolution image data and wherein the higher resolution field of view is smaller than the lower resolution field of view;

wherein said capturing lower resolution image data and said capturing higher resolution image data are performed prior to the particular time;

rendering, to a display viewable by the user, a view comprising at least part of the lower resolution image data and at least part of the higher resolution image data, wherein the rendered image is viewable by the user at the particular time.

20. The method of claim 19, wherein said determining the predictive line sight vector comprises:

determining the motion information based at least in part on output from one or more of;

one or more gaze tracking module configured to track eye movements of the user'"'"'s eyes; and

one or more accelerometers configured to track movements of the user'"'"'s head.