Reducing rendering computation and power consumption by detecting saccades and blinks

• US 10,372,205 B2
• Filed: 03/31/2016
• Issued: 08/06/2019
• Est. Priority Date: 03/31/2016
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Young, Andrew, Stafford, Jeffrey Roger
• Primary Examiner(s): Wills-Burns, Chineyere D
• Application Number:
  US15/086,953
• Publication Number:
  US 20170285735A1
• Time in Patent Office: 1,223 Days
• Field of Search: 345156, 382117
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering

Abstract

Gaze tracking data may be analyzed to determine the onset and duration of a vision interrupting event, such as a blink or saccade. Presentation of images to a viewer may then be suspended during the vision interrupting event and resumed in sufficient time to ensure that the viewer sees the image at the time the vision interrupting event has concluded.

Claims

1. A graphics processing method comprising:

obtaining gaze tracking data representing a viewer'"'"'s gaze with respect to one or more images presented to a viewer;

analyzing the gaze tracking data to determine an onset and predict a duration of a vision interrupting event wherein analyzing gaze tracking data further includes analyzing previous gaze tracking data to improve detection and prediction of duration estimation for vision interrupting events;

suspending presentation of the one or more images to the viewer during the vision interrupting event, wherein suspending presentation of the one or more images to the viewer during the vision interrupting event includes suspending graphics processing of at least one of the one or more images that would otherwise be presented to the viewer during the vision interrupting event; and

resuming presentation of the one or more images to the viewer in sufficient time to ensure that the viewer sees the one or more images at a time the vision interrupting event has concluded.

2. The method of claim 1, wherein said analyzing the gaze tracking data includes at least one of analyzing a, rotational velocity and rotational acceleration of an eye of the viewer.

3. The method of claim 1, wherein said analyzing the gaze tracking data includes analyzing information related to movement of an eyelid of the viewer.

4. The method of claim 1, wherein said suspending presentation of the one or more images to the viewer during the vision interrupting event includes turning off power to a display during the vision interrupting event.

5. The method of claim 1, wherein said suspending presentation of the one or more images to the viewer during the vision interrupting event includes turning off power to a backlight of a display during the vision interrupting event.

6. The method of claim 1, wherein said suspending presentation of the one or more images to the viewer during the vision interrupting event includes turning off power to a display during the vision interrupting event and suspending graphics processing for one or more images that would otherwise be presented on the display during the vision interrupting event.

7. The method of claim 1, wherein said suspending presentation of the one or more images to the viewer during the vision interrupting event includes avoiding transmission of data corresponding to at least one of the one or more images that would otherwise be presented during the image interrupting event.

8. The method of claim 1, wherein said resuming presentation of the one or more images includes generating foveation data representing one or more foveated images using the foveation data representing one or more regions of interest of the image determined from the gaze tracking data, wherein the one or more foveated images are characterized by a level of detail within the one or more regions of interest and a lower level of detail outside the one or more regions of interest.

9. The method of claim 8, wherein the foveation data includes data representing geometric density in the one or more regions of interest.

10. The method of claim 8, wherein the foveation data includes data representing pixel resolution in the one or more regions of interest.

11. The method of claim 8, further comprising creating a standard 2D compliant image from the foveated image data for presentation on one or more additional displays.

12. The method of claim 8, further comprising determining one or more gaze tracking parameters from the gaze tracking data;

generating adjusted foveation data representing at least one of an adjusted size and shape of one or more regions of interest in the one or more images to be subsequently presented to the viewer based on the one or more gaze tracking parameters;

generating foveation data representing one or more foveated images using the adjusted foveation data, wherein the one or more foveated images are characterized by level of detail within the one or more regions of interest and lower level of detail outside the one or more regions of interest; and

presenting the one or more foveated images with the foveated image data to the viewer.

13. The method of claim 12, wherein determining one or more gaze tracking parameters from the gaze tracking data includes determining a rate of rotation of the viewer'"'"'s eye with respect to one or more axes.

14. The method of claim 12, wherein determining one or more gaze tracking parameters from the gaze tracking data includes determining a rate of rotation of the viewer'"'"'s eye with respect to one or more axes and determining an error in a fixation of the viewer'"'"'s eye on a region of interest of the one or more regions of interest.

15. The method of claim 12, wherein determining one or more gaze tracking parameters from the gaze tracking data includes determining a rate of rotation of the viewer'"'"'s eye with respect to one or more axes and determining whether the viewer'"'"'s eye is moving in smooth pursuit.

16. The method of claim 12, wherein determining one or more gaze tracking parameters from the gaze tracking data includes determining a confidence interval regarding a current gaze position.

17. The method of claim 12, wherein determining one or more gaze tracking parameters from the gaze tracking data includes determining metrics of a blink of the viewer.

18. The method of claim 12, wherein determining one or more gaze tracking parameters from the gaze tracking data includes determining metrics of a saccade of the viewer.

19. The method of claim 12, wherein determining one or more gaze tracking parameters from the gaze tracking data includes determining a transition in a gaze direction of the viewer as a result of a change in the depth of field of the presented image.

20. The method of claim 12, wherein determining one or more gaze tracking parameters from the gaze tracking data includes determining whether the viewer is color blind.

21. The method of claim 12, wherein determining one or more gaze tracking parameters from the gaze tracking data includes determining a rate of rotation of the viewer'"'"'s eye with respect to one or more axes and determining the level of gaze stability of the viewer'"'"'s eye.

22. The method of claim 12, wherein determining one or more gaze tracking parameters from the gaze tracking data includes determining a rate of rotation of the viewer'"'"'s eye with respect to one or more axes and determining whether the viewer'"'"'s eye movement is a precursor to the viewer'"'"'s head movement.

23. The method of claim 12, wherein the adjusted foveation data includes data characterizing a particular region of interest of the one or more regions of interest, wherein the data characterizing the particular region of interest includes a foveation region and a transition region, wherein the foveation region is characterized by a higher level of detail than the transition region and the transition region is characterized by a higher level of detail than regions of a corresponding image outside the particular region of interest.

24. The method of claim 23, wherein generating adjusted foveation data representing an adjusted size and/or shape of the one or more regions of interest includes adjusting a size of the transition region.

25. The method of claim 12, wherein the adjusted foveation data includes data used to determine geometric density in the one or more regions of interest.

26. The method of claim 12, wherein the adjusted foveation data includes data used to determine pixel resolution in the one or more regions of interest.

27. The method of claim 12, wherein the adjusted foveation data includes data used to determine pixel shading quality in the one or more regions of interest.

28. The method of claim 12, further comprising creating a standard 2D compliant image from the foveated image data for presentation on one or more additional displays.

29. A system, comprising:

a processor;

a memory; and

computer-readable instructions embodied in the memory, the computer-readable instructions being configured to implement a graphics processing method when executed, the graphics processing method comprising;

obtaining gaze tracking data representing a viewer'"'"'s gaze with respect to one or more images presented to the viewer;

analyzing the gaze tracking data to determine an onset and predict a duration of a vision interrupting event, wherein analyzing gaze tracking data further includes analyzing previous gaze tracking data to improve detection and prediction of duration estimation for vision interrupting events;

suspending presentation of the one or more images to the viewer during the vision interrupting event, wherein suspending presentation of the one or more images to the viewer during the vision interrupting event includes suspending graphics processing of at least one of the one or more images that would otherwise be presented to the viewer during the vision interrupting event; and

resuming presentation of the one or more images to the viewer in sufficient time to ensure that the viewer sees the one or more images at a time the vision interrupting event has concluded.

30. A non-transitory computer-readable medium having computer-readable instructions embodied therein, the computer-readable instructions being configured to implement a graphics processing method when executed, the graphics processing method comprising:

obtaining gaze tracking data representing a viewer'"'"'s gaze with respect to one or more images presented to the viewer;

analyzing the gaze tracking data to determine an onset and predict a duration of a vision interrupting event, wherein analyzing gaze tracking data further includes analyzing previous gaze tracking data to improve detection and prediction of duration estimation for vision interrupting events;

suspending presentation of the one or more images to the viewer during the vision interrupting event, wherein suspending presentation of the one or more images to the viewer during the vision interrupting event includes suspending graphics processing of at least one of the one or more images that would otherwise be presented to the viewer during the vision interrupting event; and

resuming presentation of the one or more images to the viewer in sufficient time to ensure that the viewer sees the image at a time the vision interrupting event has concluded.