ADAPTIVE DISPLAYS USING GAZE TRACKING

• US 20120146891A1
• Filed: 12/08/2010
• Published: 06/14/2012
• Est. Priority Date: 12/08/2010
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Kalinli, Ozlem
• Granted Patent: US 8,493,390 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/426 involving on-screen location information, e.g. screen coordinates of an area at which the player is aiming with a light gun
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/1415 with means for detecting differences between the image stored in the host and the images displayed on the displays
• CPC Class Codes: G09G 2320/0261 in the context of movement of objects on the screen or movement of the observer relative to the screen
• CPC Class Codes: G09G 2340/02 Handling of images in compressed format, e.g. JPEG, MPEG
• CPC Class Codes: G09G 2340/0407 Resolution change, inclusive of the use of different resolutions for different screen areas
• CPC Class Codes: G09G 2350/00 Solving problems of bandwidth in display systems
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 2360/08 Power processing, i.e. workload management for processors involved in display operations, such as CPUs or GPUs
• CPC Class Codes: G09G 2360/10 Display system comprising arrangements, such as a coprocessor, specific for motion video images
• CPC Class Codes: G09G 2360/121 using a cache memory
• CPC Class Codes: G09G 2370/025 LAN communication management
• CPC Class Codes: G09G 5/363 Graphics controllers
• CPC Class Codes: H04N 19/115 Selection of the code volume for a coding unit prior to coding
• CPC Class Codes: H04N 19/162 User input
• CPC Class Codes: H04N 19/167 Position within a video image, e.g. region of interest [ROI]
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/503 : involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109

Abstract

Methods and systems for adapting a display screen output based on a display user'"'"'s attention. Gaze direction tracking is employed to determine a sub-region of a display screen area to which a user is attending. Display of the attended sub-region is modified relative to the remainder of the display screen, for example, by changing the quantity of data representing an object displayed within the attended sub-region relative to an object displayed in an unattended sub-region of the display screen.

Claims

1. A method of adapting a display, comprising:

presenting output on a display screen;

tracking a gaze pattern of a user viewing the display screen output;

mapping the gaze pattern to physical locations of the display screen;

determining, based on that map, a sub-region of the display screen being attended to by the user; and

modifying the display screen output by changing the quantity of data representing an object displayed within the attended sub-region relative to an object displayed in an unattended sub-region of the display screen.

2. The method of claim 1, wherein the display screen output is modified by changing the resolution of the unattended sub-region to be lower than the attended sub-region resolution or modified by changing the resolution of the attended sub-region to be greater than the unattended sub-region resolution.

3. The method of claim 1, wherein mapping the gaze pattern further comprises:

dividing the display screen into an m-by-n grid of sub-regions,

assigning a gaze pattern metric to one or more of the sub-regions; and

wherein determining the attended sub-region further comprises a comparison of the gaze pattern metric between the sub-regions.

4. The method of claim 1, further comprising:

collecting frames of image data with a video image sensor;

transmitting the collected frames of image data to the computing platform with a greater bit rate for a first portion of a collected image data frame corresponding to the attended sub-region than for a second portion of a collected image data frame corresponding to the unattended sub-region;

assembling, with a computing platform, the first and second portions of a collected image data frame into a reconstructed image data frame; and

displaying the reconstructed image data frame on the display screen.

5. The method of claim 4, further comprising applying a lower intraframe or interframe compression for the first portion of the collected image data frame than for the second portion of a collected image data frame;

or subsampling the second portion of the collected image data frame.

6. The method of claim 4, wherein the frames of image data are collected with at least an HD resolution of 1,920×

1,080 pixels, and wherein the reconstructed image data frame corresponding to the attended sub-region is displayed at the HD resolution while the unattended sub-region is displayed at a resolution below the HD resolution.

7. The method of claim 4, wherein the gaze pattern of the user is tracked based on the collected image data frames.

8. The method of claim 4, wherein the portion of the recorded frame of image data corresponding to the attended sub-region is determined by executing a contour detection algorithm on a collected image frame.

9. The method of claim 8, wherein a new contour detection algorithm is initiated either periodically or in response to receiving an indication the user blinks or a gaze shift is above a threshold.

10. The method of claim 1, wherein the gaze pattern is tracked for each of a plurality of users, wherein a sub-region of the display screen being attended to by each of the plurality of users is determined, and wherein the display screen output is modified for each attended sub-region.

11. The method of claim 10, further comprising associating a user of the plurality with an input device based on the collected image frames, and

wherein the gaze pattern tracking is performed for only the user associated with the input device.

12. A non-transistory computer-readable medium having stored thereon a set of instructions, which when executed by a processing system, cause the processing system to perform the method of claim 1.

13. An adaptive display system, comprising:

a computing platform with a display screen;

a gaze tracker to track a gaze pattern of a user viewing output from the display screen;

a gaze direction analyzer to map the gaze pattern to physical locations of the display screen and to determine, based on that map, a sub-region of the display screen being attended to by the user; and

an adaptive display processor to change the quantity of data representing an object displayed within the attended sub-region relative to an object displayed in an unattended sub-region of the display screen.

14. The system of claim 13, wherein the adaptive display processor is to modify the display screen output by changing the resolution of the unattended sub-region to be lower than the attended sub-region resolution or by changing the resolution of the attended sub-region to be greater than the unattended sub-region resolution.

15. The system of claim 13, wherein the gaze direction analyzer is to map the gaze pattern by dividing the display screen into an m-by-n grid of sub-regions, and assigning a gaze pattern metric to one or more of the sub-regions;

and

wherein the gaze direction analyzer is to determine the attended sub-region based on a comparison of the gaze pattern metric between the sub-regions.

16. The system of claim 15, wherein the gaze pattern metric is any of:

a number of fixations, a gaze pattern scan length, or a gaze pattern scan area.

17. The system of claim 13, further comprising a video image sensor to collect frames of image data,

wherein the adaptive display processor is to cause the collected frames of image data to be transmitted to the computing platform with a greater bit rate for a first portion of a collected image data frame corresponding to the attended sub-region than for a second portion of a collected image data frame corresponding to the unattended sub-region, and

wherein the computing platform is to assemble the first and second portions of a collected image data frame into a reconstructed image data frame;

and

wherein the display screen is to display the reconstructed image data frame.

18. The system of claim 17, further comprising a compression engine to apply a lower intraframe or interframe compression for the first portion of the collected image data frame than for the second portion of a collected image data frame;

or wherein the adaptive display processor is to subsample the second portion of the collected image data frame.

19. The system of claim 17, wherein the video image sensor is to collect the frames of image data with at least an HD resolution of 1,920×

1,080 pixels, and wherein the display screen is to display the reconstructed image data frame corresponding to the attended sub-region at the HD resolution and the unattended sub-region at a resolution below the HD resolution.

20. The system of claim 13, wherein the gaze tracker is to track the gaze pattern of the user based on the collected image data frames.

21. The system of claim 13, wherein the gaze direction analyzer is to determine the portion of the recorded frame of image data corresponding to the attended sub-region by executing a contour detection algorithm on a collected image frame.

22. The system of claim 21, wherein the gaze direction analyzer is to initiate a new contour detection algorithm either periodically or in response to receiving an indication the user blinks or a gaze shift is above a threshold.

23. The system of claim 13, wherein the gaze tracker is to track a gaze pattern for each of a plurality of users, wherein the gaze direction analyzer is to determine a sub-region of the display screen being attended to by each of the plurality of users.

24. The system of claim 23, further comprising a user input device identifiable by the computing platform, and wherein the gaze tracker is to track a gaze pattern of only a user associated, based on the collected image frames, with the input device.