System and method of gaze predictive rendering of a focal area of an animation

• US 10,255,714 B2
• Filed: 08/24/2016
• Issued: 04/09/2019
• Est. Priority Date: 08/24/2016
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Mitchell, Kenneth J., Andrews, Sheldon, Cosker, Darren, Swafford, Nicholas T.
• Primary Examiner(s): Poon, King
• Assistant Examiner(s): Peren, Vincent
• Application Number:
  US15/245,523
• Publication Number:
  US 20180061116A1
• Time in Patent Office: 958 Days
• Field of Search: 345428
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 13/20 3D [Three Dimensional] animation
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G09G 2340/0407 Resolution change, inclusive of the use of different resolutions for different screen areas
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T

Abstract

Individual images for individual frames of an animation may be rendered to include individual focal areas. A focal area may include one or more of a foveal region corresponding to a gaze direction of a user, an area surrounding the foveal region, and/or other components. The foveal region may comprise a region along the user'"'"'s line of sight that permits high visual acuity with respect to a periphery of the line of sight. A focal area within an image may be rendered based on parameter values of rendering parameters that are different from parameter values for an area outside the focal area.

Claims

1. A system configured for gaze-predictive rendering of a focal area of an animation presented on a display, wherein the animation includes a sequence of frames, the sequence of frames including a first frame and a plurality of subsequent frames, the system comprising:

one or more physical processors configured by machine-readable instructions to;

obtain state information describing a state of a virtual space, the state at an individual point in time defining one or more virtual objects within the virtual space and their positions;

determine a field of view of the virtual space, the frames of the animation being images of the virtual space within the field of view, such that the first frame is an image of the virtual space within the field of view at a point in time that corresponds to the first frame;

apply the state information for a subsequent frame in the plurality of subsequent frames for the animation to a machine learning model to generate a prediction, the prediction relating to one or more expected saccades and one or more gaze directions of a user currently viewing the presented animation, wherein the machine learning model is trained based on statistical targets of eye fixation corresponding to the user'"'"'s foveal region, and wherein the statistical targets of eye fixation are precomputed on a database of previous eye tracked viewing sessions of the animation;

determine, prior to rendering the subsequent frame, the focal area of the subsequent frame within the field of view based on the prediction, such that the focal area includes a foveal region corresponding to the one or more expected saccades and the one or more gaze directions and includes one or more regions outside of the foveal region, wherein the foveal region is a region along the user'"'"'s line of sight that permits high visual acuity with respect to a periphery of the line of sight; and

render, from the state information, one or more images for the subsequent frame of the animation, the one or more images depicting the virtual space within the field of view determined at individual points in time, wherein an area outside of the focal area of the subsequent frame is rendered at a lower resolution than that of the focal area to reduce latency.

2. The system of claim 1, wherein determining the focal area comprises determining a size of the focal area, the size being a function of one or more of a system latency, the user'"'"'s maximum saccadic speed, the user'"'"'s distance from the display, an angle subtended by a fovea of the user'"'"'s eye, or a pixel density of the display.

3. The system of claim 1, wherein determining the focal area comprises determining a position of a center point of the focal area based on the predicted gaze direction.

4. The system of claim 1, wherein the statistical targets are further determined based on the positions of the one or more virtual objects within the virtual space defined by the state information.

5. The system of claim 1, wherein the statistical targets are determined by tracking, via a gaze tracking device, gaze direction of one or more users viewing the animation and/or other animations.

6. The system of claim 5, wherein the one or more users comprise the user viewing the animation.

7. The system of claim 1, wherein determining the field of view of the virtual space further comprises determining an orientation of the user.

8. The system of claim 7, further comprising a virtual reality headset, and wherein:

the virtual reality headset includes the display; and

determining the orientation of the user comprises determining an orientation of the virtual reality headset.

9. The system of claim 1, wherein the statistical targets are determined based on roles previously assigned to the individual ones of the one or more virtual objects within the virtual space defined by the state information, and wherein an individual role comprises one or more of a player character, a teammate, a main or central character, a minor character, a protagonist, an antagonist, an anti-hero, an enemy, a combatant, a speaker, or a listener.

10. A method of gaze-predictive rendering of a focal area of an animation presented on a display, the animation including a sequence of frames, the sequence of frames including a first frame and a plurality of subsequent frames, the method being implemented in a computer system comprising one or more physical processors and storage media storing machine-readable instructions, the method comprising:

obtaining state information describing state of a virtual space, the state at an individual point in time defining one or more virtual objects within the virtual space and their positions;

determining a field of view of the virtual space, the frames of the animation being images of the virtual space within the field of view, such that the first frame is an image of the virtual space within the field of view at a point in time that corresponds to the first frame;

applying the state information for a subsequent frame in the plurality of subsequent frames for the animation to a machine learning model to generate a prediction, the prediction relating to one or more expected saccades and one or more gaze directions of a user currently viewing the presented animation, wherein the machine learning model is trained based on statistical targets of eye fixation corresponding to the user'"'"'s foveal region, and wherein the statistical targets of eye fixation are precomputed on a database of previous eye tracked viewing sessions of the animation;

determining, prior to rendering the subsequent frame, the focal area of the subsequent frame within the field of view based on the prediction, such that the focal area includes a foveal region corresponding to the one or more expected saccades and the one or more gaze directions and includes one or more regions outside of the foveal region, wherein the foveal region is a region along the user'"'"'s line of sight that permits high visual acuity with respect to a periphery of the line of sight; and

rendering from the state information, one or more images for the subsequent frame of the animation, the one or more images depicting the virtual space within the field of view determined at individual points in time, wherein an area outside of the focal area of the subsequent frame is rendered at a lower resolution than that of the focal area to reduce latency.

11. The method of claim 10, wherein determining the focal area comprises determining a size of the focal area, the size being a function of one or more of a system latency, the user'"'"'s maximum saccadic speed, the user'"'"'s distance from the display, an angle subtended by a fovea of the user'"'"'s eye, or a pixel density of the display.

12. The method of claim 10, wherein determining the focal area comprises determining a position of a center point of the focal area based on the predicted gaze direction.

13. The method of claim 10, wherein the statistical targets are further determined based on the positions of the one or more virtual objects within the virtual space defined by the state information.

14. The method of claim 10, wherein the statistical targets are determined by tracking gaze direction of one or more users viewing the animation and/or other animations.

15. The method of claim 14, wherein the one or more users comprise the user viewing the animation.

16. The method of claim 10, wherein determining the field of view of the virtual space further comprises determining an orientation of the user.

17. The method of claim 16, wherein determining an orientation of the user comprises determining an orientation of a virtual reality headset worn by the user.

18. The method of claim 10, wherein the statistical targets are determined based on roles previously assigned to the individual ones of the one or more virtual objects within the virtual space defined by the state information, and wherein an individual role comprises one or more of a player character, a teammate, a main or central character, a minor character, a protagonist, an antagonist, an anti-hero, an enemy, a combatant, a speaker, or a listener.