Apparatus and method for dynamic graphics rendering based on saccade detection

• US 10,585,475 B2
• Filed: 12/22/2017
• Issued: 03/10/2020
• Est. Priority Date: 09/04/2015
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Mallinson, Dominic Saul
• Primary Examiner(s): Sharif-Tafreshi, Koosha
• Application Number:
  US15/853,529
• Publication Number:
  US 20180136722A1
• Time in Patent Office: 809 Days
• Field of Search: 345209, 345633, 345 8
• CPC Class Codes: G06F 1/3231 Monitoring the presence, absence or movement of users
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: H04N 21/234345 the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

A method for rendering computer graphics based on saccade detection is provided. One embodiment of the method includes rendering a computer simulated scene for display to a user, detecting an onset of a saccade that causes saccadic masking in an eye movement of the user viewing the computer simulated scene, and reducing a computing resource used for rendering frames of the computer simulated scene during at least a portion of a duration of the saccade. Systems perform similar steps, and non-transitory computer readable storage mediums each storing one or more computer programs are also provided.

Claims

1. A method, comprising:

detecting a saccade in an eye of a user viewing a computer simulation;

predicting, prior to an end of the saccade, where a fixation point of the user will be located in a computer simulated scene at the end of the saccade; and

rendering the predicted fixation point in a greater amount of detail than other portions of the computer simulated scene;

wherein the detecting a saccade in an eye of a user viewing a computer simulation comprises;

detecting an onset of the saccade prior to a start of saccadic movement of the eye.

2. The method of claim 1, wherein the rendering the predicted fixation point in a greater amount of detail than other portions of the computer simulated scene comprises:

rendering the predicted fixation point at a higher resolution than other portions of the computer simulated scene.

3. The method of claim 1, wherein the predicting where a fixation point of the user will be located in a computer simulated scene at the end of the saccade comprises:

predicting a future gaze location of the user.

4. The method of claim 1, wherein the predicting where a fixation point of the user will be located in a computer simulated scene at the end of the saccade comprises:

predicting a duration of the saccade.

5. The method of claim 4, wherein the predicting the duration of the saccade is based on at least one of statistical data relating to durations of saccades in human eyes and a magnitude of a measurement used to detect the onset of the saccade.

6. The method of claim 4, further comprising:

detecting the end of the saccade; and

calculating the duration of the saccade based on at least the detected end of the saccade.

7. The method of claim 6, further comprising:

comparing the calculated duration of the saccade to the predicted duration of the saccade; and

using results of the comparison to adjust a model used to predict the duration of the saccade.

8. The method of claim 1, wherein the saccade is detected using one or more of an optical sensor, an optical flow sensor, an electromyography (EMG) sensor, a position sensing device, and a Doppler effect sensor.

9. The method of claim 1, further comprising:

calibrating an individualized eye movement model for the user viewing the computer simulation, wherein the detecting the saccade is based on the individualized eye movement model.

10. The method of claim 1, further comprising:

reducing a computing resource used for rendering frames of the computer simulation during at least a portion of a duration of the saccade.

11. The method of claim 10, further comprising:

determining a confidence level of the detection of the saccade.

12. The method of claim 11, further comprising:

determining an amount of the computing resource to reduce based on the confidence level.

13. The method of claim 11, further comprising:

selecting from two or more methods of reducing the computing resource used for rendering frames of the computer simulation based on the confidence level.

14. The method of claim 10, further comprising:

reallocating the computing resource to perform low frequency update calculations for rendering frames of the computer simulation.

15. The method of claim 14, wherein the low frequency update calculations comprise lighting calculations.

16. A system comprising:

an eye movement sensor;

a display device for displaying computer simulated scenes to a user; and

a processor communicatively coupled to the eye movement sensor and the display device, the processor being configured to execute steps comprising;

detecting a saccade in an eye of the user viewing the computer simulated scenes;

predicting, prior to an end of the saccade, where a fixation point of the user will be located in a computer simulated scene at the end of the saccade; and

rendering the predicted fixation point in a greater amount of detail than other portions of the computer simulated scene;

wherein the detecting a saccade in an eye of the user viewing the computer simulated scenes comprises;

detecting an onset of the saccade prior to a start of saccadic movement of the eye.

17. The system of claim 16, wherein the rendering the predicted fixation point in a greater amount of detail than other portions of the computer simulated scene comprises:

rendering the predicted fixation point at a higher resolution than other portions of the computer simulated scene.

18. The system of claim 16, wherein the predicting where a fixation point of the user will be located in a computer simulated scene at the end of the saccade comprises:

predicting a future gaze location of the user.

19. The system of claim 16, wherein the predicting where a fixation point of the user will be located in a computer simulated scene at the end of the saccade comprises:

predicting a duration of the saccade.

20. The system of claim 16, wherein the eye movement sensor comprises one or more of an optical sensor, an optical flow sensor, an electromyography (EMG) sensor, a position sensing device, and a Doppler effect sensor.

21. The system of claim 16, wherein the processor is further configured to execute steps comprising:

calibrating an individualized eye movement model for the user viewing the computer simulated scenes, wherein the detecting the saccade is based on the individualized eye movement model.

22. The system of claim 16, wherein the processor is further configured to execute steps comprising:

reducing a computing resource used for rendering frames of the computer simulated scenes during at least a portion of a duration of the saccade.

23. The system of claim 22, wherein the processor is further configured to execute steps comprising:

determining a confidence level of the detection of the saccade.

24. The system of claim 23, wherein the processor is further configured to execute steps comprising:

determining an amount of the computing resource to reduce based on the confidence level.

25. The system of claim 23, wherein the processor is further configured to execute steps comprising:

selecting from two or more methods of reducing the computing resource used for rendering frames of the computer simulated scenes based on the confidence level.

26. A non-transitory computer readable storage medium storing one or more programs that cause a processor based system to execute steps comprising:

detecting a saccade in an eye of a user viewing a computer simulation;

predicting, prior to an end of the saccade, where a fixation point of the user will be located in a computer simulated scene at the end of the saccade; and

rendering the predicted fixation point in a greater amount of detail than other portions of the computer simulated scene;

wherein the detecting a saccade in an eye of a user viewing a computer simulation comprises;

detecting an onset of the saccade prior to a start of saccadic movement of the eye.

27. The non-transitory computer readable storage medium of claim 26, wherein the rendering the predicted fixation point in a greater amount of detail than other portions of the computer simulated scene comprises:

rendering the predicted fixation point at a higher resolution than other portions of the computer simulated scene.

28. The non-transitory computer readable storage medium of claim 26, wherein the predicting where a fixation point of the user will be located in a computer simulated scene at the end of the saccade comprises:

predicting a future gaze location of the user.

29. The non-transitory computer readable storage medium of claim 26, wherein the predicting where a fixation point of the user will be located in a computer simulated scene at the end of the saccade comprises:

predicting a duration of the saccade.

30. The non-transitory computer readable storage medium of claim 26, wherein the one or more programs are further configured to cause the processor based system to execute steps comprising:

reducing a computing resource used for rendering frames of the computer simulation during at least a portion of a duration of the saccade.

31. The non-transitory computer readable storage medium of claim 30, wherein the one or more programs are further configured to cause the processor based system to execute steps comprising:

determining a confidence level of the detection of the saccade.

32. The non-transitory computer readable storage medium of claim 31, wherein the one or more programs are further configured to cause the processor based system to execute steps comprising:

selecting from two or more methods of reducing the computing resource used for rendering frames of the computer simulation based on the confidence level.