ADAPTIVE PARAMETERS IN IMAGE REGIONS BASED ON EYE TRACKING INFORMATION

• US 20190318708A1
• Filed: 06/26/2019
• Published: 10/17/2019
• Est. Priority Date: 08/01/2016
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Facebook Technologies, LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Tall, Martin Henrik, San Agustin Lopez, Javier, Dahl, Rasmus
• Granted Patent: US 10,984,756 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06T 9/00 Image coding bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00
• CPC Class Codes: G09G 2320/0693 Calibration of display systems
• CPC Class Codes: G09G 2340/04 Changes in size, position or resolution of an image
• CPC Class Codes: G09G 2340/0407 Resolution change, inclusive of the use of different resolutions for different screen areas
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: G09G 5/37 Details of the operation on graphic patterns G09G5/38 takes precedence
• CPC Class Codes: G09G 5/391 Resolution modifying circuits, e.g. variable screen formats
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/162 User input
• CPC Class Codes: H04N 19/167 Position within a video image, e.g. region of interest [ROI]
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object

Abstract

A display system divides a screen into regions and applies a different set of rendering/encoding parameters to each region. The system applies a first set of parameters to a first region that is being viewed by a fovea of an eye of a user. The system may also apply a second set of parameters to a second region that is being viewed by a parafovea of the eye, and apply a third set of parameters to a third region that is being viewed by the area of the eye outside of the parafovea. The first set of parameters are selected to yield relatively high image quality, while the second set of parameters are yield intermediate quality, and the third set of parameters yield lower quality. As a result, the second region and the third region can be rendered, encoded, and transmitted with less computing power and less bandwidth.

Claims

1. A method comprising:

determining a plurality of screen regions of a screen of a display device, the plurality of screen regions comprising a first screen region separate from a second screen region, the first screen region containing a point of regard of a user, wherein the second screen region surrounds the first screen region;

encoding a portion of an image corresponding to the first screen region to be displayed on the screen with a first image quality;

encoding another portion of the image corresponding to the second screen region with a second image quality, the first image quality higher than the second image quality; and

transmitting the encoded image to the display device to be displayed on the screen,

wherein a size of the first screen region and a size of the second screen region are based on one or more predetermined parameters.

2. The method of claim 1, wherein the first screen region is a circle centered on the point of regard, and the second screen region is an annulus centered around the point of regard.

3. The method of claim 2, wherein the one or more predetermined parameters comprise:

a radius of the first screen region, and

a radius of an outer circle of the annulus, the outer circle corresponding to an outer edge of the second screen region.

4. The method of claim 1, wherein

the first screen region is determined based on an intersection of a first cone with the screen of the display device, and

the second screen region is determined based on an intersection of a second cone with the screen of the display device.

5. The method of claim 4, wherein the one or more predetermined parameters comprise:

a first angle of the first cone, wherein a conical surface of the first cone is offset from an axis by the first angle, and

a second angle of the second cone, wherein a conical surface of the second cone is offset from the axis by the second angle.

6. The method of claim 4, further comprising determining a gaze vector based on eye characteristics of the user, wherein an axis of the first cone and of the second cone is defined by the determined gaze vector.

7. The method of claim 1, wherein the first screen region is a rectangle centered on the point of regard, and a shape of the second screen region is defined by an outer rectangle centered on the point of regard, and the one or more predetermined parameters comprise:

a length of the first screen region, and

a length of a side of the outer rectangle that defines the shape of the second screen region.

8. The method of claim 7, wherein each side of the first screen region has a same length, and each side of the outer rectangle has a same length.

9. The method of claim 1, further comprising:

determining one or more additional screen regions, the one or more additional screen regions configured to provide a smoother transition between the first screen region and the second screen region; and

updating a size of each of the first screen region, the second screen region, and the one or more additional screen regions based on at least one of a noise in eye characteristics, an estimated accuracy of the eye characteristics, and a direction of gaze vectors.

10. The method of claim 9, further comprising:

responsive to detecting a rapid eye movement between fixation points, increasing the size of each of the first screen region, the second screen region, and the one or more additional screen regions; and

responsive to detecting that a fixation is stable after detecting the rapid eye movement, reducing the size of each of the first screen region, the second screen region, and the one or more additional screen regions.

11. The method of claim 1, wherein the first screen region corresponds to a region of the screen viewed by a fovea of an eye of the user, and the second screen region corresponds to a region of the screen viewed by a parafovea of an eye of the user.

12. The method of claim 1, wherein the first image quality comprises a first image resolution, the second image quality comprises a second image resolution, and the first image resolution is higher than the second image resolution.

13. The method of claim 1, wherein the first image quality comprises a first framerate, the second image quality comprises a second framerate, and the first framerate is higher than the second framerate.

14. A non-transitory computer-readable medium storing instructions that, when executed by one or more processor, cause the one or more processors to perform operations comprising:

determining a plurality of screen regions of a screen of a display device, the plurality of screen regions comprising a first screen region separate from a second screen region, the first screen region containing a point of regard of a user, wherein the second screen region surrounds the first screen region;

encoding a portion of an image corresponding to the first screen region to be displayed on the screen with a first image quality;

encoding another portion of the image corresponding to the second screen region with a second image quality, the first image quality higher than the second image quality; and

transmitting the encoded image to the display device to be displayed on the screen,

wherein a size of the first screen region and a size of the second screen region are based on one or more predetermined parameters.

15. The non-transitory computer-readable medium of claim 14, wherein the first screen region corresponds to a region of the screen viewed by a fovea of an eye of the user, and the second screen region corresponds to a region of the screen viewed by a parafovea of an eye of the user.

16. The non-transitory computer-readable medium of claim 14, wherein the first image quality comprises a first image resolution, the second image quality comprises a second image resolution, and the first image resolution is higher than the second image resolution.

17. The non-transitory computer-readable medium of claim 14, wherein the first image quality comprises a first framerate, the second image quality comprises a second framerate, and the first framerate is higher than the second framerate.

18. The non-transitory computer-readable medium of claim 14, wherein the first screen region is a circle centered on the point of regard, and the second screen region is an annulus centered around the point of regard.

19. The non-transitory computer-readable medium of claim 14, wherein

the first screen region is determined based on an intersection of a first cone with the screen of the display device, and

the second screen region is determined based on an intersection of a second cone with the screen of the display device.

20. A system comprising:

one or more processors; and

a non-transitory computer-readable medium storing instructions that, when executed by the one or more processor, cause the one or more processors to perform operations comprising;

determining a plurality of screen regions of a screen of a display device, the plurality of screen regions comprising a first screen region separate from a second screen region, the first screen region containing a point of regard of a user, wherein the second screen region surrounds the first screen region;

encoding a portion of an image corresponding to the first screen region to be displayed on the screen with a first image quality;

encoding another portion of the image corresponding to the second screen region with a second image quality, the first image quality higher than the second image quality; and

transmitting the encoded image to the display device to be displayed on the screen,

wherein a size of the first screen region and a size of the second screen region are based on one or more predetermined parameters.