Adaptive displays using gaze tracking

• US 8,493,390 B2
• Filed: 12/08/2010
• Issued: 07/23/2013
• Est. Priority Date: 12/08/2010
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment America (Sony Group Corp.)
• Inventors: Kalinli, Ozlem
• Primary Examiner(s): Nguyen, Hau
• Assistant Examiner(s): Gray, Ryan M
• Application Number:
  US12/963,100
• Publication Number:
  US 20120146891A1
• Time in Patent Office: 958 Days
• Field of Search: None
• US Class Current: 345/428
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/426 involving on-screen location information, e.g. screen coordinates of an area at which the player is aiming with a light gun
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/1415 with means for detecting differences between the image stored in the host and the images displayed on the displays
• CPC Class Codes: G09G 2320/0261 in the context of movement of objects on the screen or movement of the observer relative to the screen
• CPC Class Codes: G09G 2340/02 Handling of images in compressed format, e.g. JPEG, MPEG
• CPC Class Codes: G09G 2340/0407 Resolution change, inclusive of the use of different resolutions for different screen areas
• CPC Class Codes: G09G 2350/00 Solving problems of bandwidth in display systems
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 2360/08 Power processing, i.e. workload management for processors involved in display operations, such as CPUs or GPUs
• CPC Class Codes: G09G 2360/10 Display system comprising arrangements, such as a coprocessor, specific for motion video images
• CPC Class Codes: G09G 2360/121 using a cache memory
• CPC Class Codes: G09G 2370/025 LAN communication management
• CPC Class Codes: G09G 5/363 Graphics controllers
• CPC Class Codes: H04N 19/115 Selection of the code volume for a coding unit prior to coding
• CPC Class Codes: H04N 19/162 User input
• CPC Class Codes: H04N 19/167 Position within a video image, e.g. region of interest [ROI]
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/503 : involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109

Abstract

Methods and systems for adapting a display screen output based on a display user'"'"'s attention. Gaze direction tracking is employed to determine a sub-region of a display screen area to which a user is attending. Display of the attended sub-region is modified relative to the remainder of the display screen, for example, by changing the quantity of data representing an object displayed within the attended sub-region relative to an object displayed in an unattended sub-region of the display screen.

Claims

1. A method of adapting a display, comprising:

collecting image data at a frame rate with a video image sensor;

presenting output on a display screen, the output including one or more frames of the image data;

tracking a gaze pattern of a user viewing the display screen output;

mapping the gaze pattern to physical locations of the display screen;

determining, based on the mapping, a sub-region of the display screen being attended to by the user, the determining performed while collecting additional frames of image data with the video sensor, and performed periodically at a frequency below the frame rate, or in response to an indication the user blinked, or a gaze shift is above a threshold;

transmitting the additional frames of the image data to the computing platform with a greater bit rate for a first portion of each additional frame of the image data corresponding to the attended sub-region than for a second portion of each additional frame of the image data corresponding to the unattended sub-region;

assembling, with a computing platform, the first and second portions of the additional frames into reconstructed image data frames; and

updating the output on the display screen with the reconstructed image data frames.

2. The method of claim 1, wherein the display screen output is modified by changing the resolution of the unattended sub-region to be lower than the attended sub-region resolution or modified by changing the resolution of the attended sub-region to be greater than the unattended sub-region resolution.

3. The method of claim 1, wherein mapping the gaze pattern further comprises:

dividing the display screen into an m-by-n grid of sub-regions,

assigning a gaze pattern metric to one or more of the sub-regions; and

wherein determining the attended sub-region further comprises a comparison of the gaze pattern metric between the sub-regions.

4. The method of claim 1, wherein transmitting the additional frames of the image data to the computing platform further comprises applying a lower intraframe or interframe compression for the first portion of the collected image data frame than for the second portion of a collected image data frame;

or subsampling the second portion of the collected image data frame.

5. The method of claim 1, wherein the frames of image data are collected with at least an HD resolution of 1,920×

1,080 pixels, and wherein the reconstructed image data frame corresponding to the attended sub-region is displayed at the HD resolution while the unattended sub-region is displayed at a resolution below the HD resolution.

6. The method of claim 1, wherein the collected image data frames include the user, wherein the gaze pattern of the user is tracked based on the collected image data frames.

7. The method of claim 1, wherein the portion of the recorded frame of image data corresponding to the attended sub-region is determined by executing a contour detection algorithm on a collected image frame.

8. The method of claim 1, wherein updating the output on the display screen further comprises displaying with the reconstructed image data frames a graphical object rendered by the computing platform.

9. The method of claim 1, wherein the video image sensor has a field of view (FOV) encompassing a plurality of users, wherein the gaze pattern is tracked for each of the plurality of users, wherein a sub-region of the display screen being attended to by each of the plurality of users is determined, and wherein the display screen output is modified for each attended sub-region.

10. The method of claim 9, further comprising associating a user of the plurality with an input device based on the collected image frames, and

wherein the gaze pattern tracking is performed for only the user associated with the input device.

11. A non-transitory computer-readable medium having stored thereon a set of instructions, which when executed by a processing system, cause the processing system to perform a method comprising:

collecting image data with a video image sensor at a frame rate;

presenting output on a display screen, the output including one or more frames of the image data;

tracking a gaze pattern of a user viewing the display screen output;

mapping the gaze pattern to physical locations of the display screen;

determining, based on the mapping, a sub-region of the display screen being attended to by the user, the determining performed while collecting additional frames of image data with the video sensor, and performed periodically at a frequency below the frame rate, or in response to an indication the user blinked, or a gaze shift is above a threshold;

transmitting the additional frames of the image data to the computing platform with a greater bit rate for a first portion of each additional frame of the image data corresponding to the attended sub-region than for a second portion of each additional frame of the image data corresponding to the unattended sub-region;

assembling, with a computing platform, the first and second portions of the additional frames into reconstructed image data frames; and

updating the output on the display screen with the reconstructed image data frames.

12. An adaptive display system, comprising:

a video image sensor to collect image data at a frame rate;

a computing platform with a display screen to output one or more frames of the image data;

a gaze tracker to track a gaze pattern of a user viewing output from the display screen;

a gaze direction analyzer to map the gaze pattern to physical locations of the display screen and to determine, based on that map, a sub-region of the display screen being attended to by the user, wherein the determining is to be performed while collecting additional frames of image data with the video sensor, and performed periodically at a frequency below the frame rate, or in response to an indication the user blinked, or a gaze shift is above a threshold; and

an adaptive display processor to change the quantity of data representing an object displayed within the attended sub-region relative to an object displayed in an unattended sub-region of the display screen, wherein the adaptive display processor is further to cause the additional frames of image data to be transmitted to the computing platform with a greater bit rate for a first portion of each additional frame of the image data corresponding to the attended sub-region than for a second portion of each additional frame of the image data corresponding to the unattended sub-region, and

wherein the computing platform is to assemble the first and second portions of the additional frames into reconstructed image data frames, and is to update the output on the display screen with the reconstructed image data frames.

13. The system of claim 12, wherein the adaptive display processor is to modify the display screen output by changing the resolution of the unattended sub-region to be lower than the attended sub-region resolution or by changing the resolution of the attended sub-region to be greater than the unattended sub-region resolution.

14. The system of claim 12, wherein the gaze direction analyzer is to map the gaze pattern by dividing the display screen into an m-by-n grid of sub-regions, and assigning a gaze pattern metric to one or more of the sub-regions;

and

wherein the gaze direction analyzer is to determine the attended sub-region based on a comparison of the gaze pattern metric between the sub-regions.

15. The system of claim 14, wherein the gaze pattern metric is any of:

a number of fixations, a gaze pattern scan length, or a gaze pattern scan area.

16. The system of claim 12, further comprising a compression engine to apply a lower intraframe or interframe compression for the first portion of the collected image data frame than for the second portion of a collected image data frame;

or wherein the adaptive display processor is to subsample the second portion of the collected image data frame.

17. The system of claim 12, wherein the video image sensor is to collect the frames of image data with at least an HD resolution of 1,920×

1,080 pixels, and wherein the display screen is to display the reconstructed image data frame corresponding to the attended sub-region at the HD resolution and the unattended sub-region at a resolution below the HD resolution.

18. The system of claim 12, wherein the collected image data frames include the user, and wherein the gaze tracker is to track the gaze pattern of the user based on the collected image data frames.

19. The system of claim 12, wherein the gaze direction analyzer is to determine the portion of the recorded frame of image data corresponding to the attended sub-region by executing a contour detection algorithm on a collected image frame.

20. The system of claim 12, wherein the computing platform is further to render a graphical object for display with the reconstructed image data frames.

21. The system of claim 12, wherein the gaze tracker is to track a gaze pattern for each of a plurality of users, wherein the gaze direction analyzer is to determine a sub-region of the display screen being attended to by each of the plurality of users.

22. The system of claim 21, further comprising a user input device identifiable by the computing platform, and wherein the gaze tracker is to track a gaze pattern of only a user associated, based on the collected image frames, with the input device.